Paige Niedringhaus RSS feed

Getting the Most out of Claude Code

Sat, 06 Dec 2025 00:00:00 GMT

</figure>

Introduction

When AI coding assistants first emerged a few years ago they were less than impressive for doing much beyond basic autocomplete tasks.

But then things started to pick up in late 2023 and 2024 when GitHub Copilot Chat hit the scene, Devin AI caused a stir, and finally, Claude and Claude Code began to really win the hearts and minds of web developers and software engineers of all stripes.

Like many of you, I was skeptical of the AI agents at first. GitHub Copilot Chat many times, failed to grasp the basic context or even understand simple code patterns within the file it was told to modify, Devin was eager but again, was ham fisted in its actual code implementation (and difficult to guide via Slack conversation threads), and my initial attempts with Claude resulted in lost time, tokens, and unwieldy amounts of spaghetti code that I ultimately threw away to start over.

Over the past 6+ months, however, I've been working with Claude Code, Anthropic's terminal-based AI coding assistant (and believe me, I've never been a terminal dev, IDEs like VS Code are my coding happy place), and learned how to be much more effective when dealing with Claude.

In this article I'm going to share with you the techniques I've learned to get the most out of Claude Code and deliver high quality code and features in shorter amounts of time, because like it or not, AI-assisted coding is the future for us all.

Claude Code

The terminal has never been my favorite. I know enough bash to be dangerous and have my go to git commands memorized by heart, but give me a feature-rich, extension-packed IDE like VS Code any day, and I'm off to the races (which probably makes sense since the parts of web development I enjoy most are frontend-focused features).

But Claude Code began life as a terminal-only CLI, and I have to say I've gotten quite used to having it open and doing its thing in one terminal window (I use Warp as my preferred terminal), and VS Code open at the same time so I can follow along with the file changes taking place, run the dev server or tests myself, and point out linting and TypeScript errors that Claude might be ignoring or glossing over in an attempt to always be right the first time.

NOTE: Since Claude Code debuted, the team at Anthropic have also released native extensions for Claude Code so it can run inside your preferred IDE of choice, but I have to say, after trying the VS Code extension of Claude, I didn't like it as much as I like having Claude in a totally separate terminal window.

The extension was a bit buggy for me (unusual crashes or getting stuck in thinking loops until I quit the IDE and restarted), and while I'm sure its reliability has improved, it seemed like the responses weren't quite as good as what I was coming to expect while working with Claude Code in the terminal. That's just my opinion though, I have no verifiable proof to back up this impression.

Below, I've tried to group the things I've learned about getting the most out of Claude Code into broad categories. This isn't an exhaustive list, but a combination of some (or all) of these tips helps me enjoy the productivity boost Claude can provide for my code, while still feeling in control of the final code outcome. I hope it helps you to dial in your own use of Claude as well, because it really can be quite the coding boost when used effectively.

NOTE: Throughout this article I'll interchangeably use the terms "Claude" and "Claude Code", but know that with each I am referring to the same thing: the Claude Code CLI that runs in a terminal, not the Claude Desktop program that also exists.

Context matters: give Claude plenty of it

Let's start with context, because when Claude Code has context of the problem it's trying to solve, and the existing codebase it's working within, its solutions will almost guaranteed be better.

Add CLAUDE.md files to your project

CLAUDE.md files are special files you can add to a project that Claude will automatically pull into its context when starting a conversation.

Good things to include in this file are things like:

General project overview, architecture, and tech stack
Common development/bash commands and development workflows
Code style guidelines
Testing instructions
Other info you want Claude to remember or be aware of

Providing all of this inside a markdown file for Claude, makes it easy to review, revise, and update as needed, and by checking it in as part of your repo, it means your whole team can give their Claude Code agents the same basic starting point whenever they begin a new session with Claude's help.

These files can live in the root of a repo, inside specific folders within a repo (like if you have a monorepo), or even in your machine's home folder (which means it will apply to all Claude sessions everywhere).

To make a new CLAUDE.md file for a project run the /init command inside of Claude Code and it will generate a CLAUDE.md file for you that you can then refine over time.

For example, in one of my projects, there's a CLAUDE.md file at the root of the project explaining the basic overview of the app and what it does, and CLAUDE.md files inside both the backend/ and frontend/ folders explaining the tech stacks, important npm commands, testing practices, code organization, and PR etiquette among other things.

When you're writing these files just keep them succinct and easily human-readable, and you should be starting from a good place.

Don't be afraid to have Claude re-review this file from time to time either, and update it as the project continues to evolve - it's meant to be a living document.

Copy paste all the things to Claude

They say a picture is worth a thousand words and once I learned I could simply paste all sorts of things into Claude Code's terminal, I began using it liberally to give Claude more context.

I regularly take screenshots of what I'm seeing while manually testing Claude's implementation in the browser to show Claude weird styling and layout issues, console feedback and errors, or code that's just not behaving the way it's supposed to, and paste it into Claude's terminal (there's a better way to debug this sort of thing with Claude Code that I'll cover later in the article). Likewise, if VS Code is lighting up like a Christmas tree with TypeScript or linting errors or my GitHub Action workflows are failing in GitHub, I'll copy the text from those and paste that into Claude. Or if there's a web page that I want Claude to read for additional info or examples of what I'm trying to achieve, I'll paste that in and instruct Claude to visit it.

Claude has proven pretty adept at being able to deduce possible issues and solutions just from screenshots, so it's definitely worth your time to feed some images to Claude with a short comment about where the screenshot came from or what it pertains to to see if Claude can pinpoint the problem.

Use the /resume slash command to rehydrate previous conversations

Oftentimes, I'll finish a new feature, open a pull request, and get it reviewed and merged in, only to pick up another feature that builds off the one I was working on previously. When this happens, using Claude's /resume command allows me to go back through previous conversations I've had with Claude and continue one of those conversations.

In this way (assuming the context window isn't already full and the conversation has to be compacted immediately), Claude already has prior context to draw from as I instruct it in how to extend the new feature I want.

When you use the /resume command after starting up Claude Code in the terminal, Claude will present a list of previous sessions to choose from that happened within that repo, along with details like when the conversation happened, how many messages were in the conversation, and even the name of the feature branch the conversation took place in, which makes finding the right conversation to resume from easier.

</figure>

NOTE: This can be extra helpful when you may have worked on other things in between related features and need to reacquaint yourself with the code. Claude can give you a quick summary of a conversation after choosing to resume it to help get you back in the right mental model faster.

Keep Claude's sessions focused

I know I just said "give Claude lots of context," but at the same time, keep sessions (and context) focused. What I mean when I say this is don't use the same session with Claude for working on different tickets or features.

Claude has enough trouble staying on track with one feature, especially when you start to factor in things like writing or updating unit tests, fixing TypeScript or linting errors, adhering to the project's style guide lines or documentation requirements, etc. Throwing Claude a curveball in the middle of a session is as disruptive to it as when someone sends you an email or pings you on Slack when you're in the middle of deep thinking and your mental model is locked in.

Also, previous conversations in the same session just serve to distract Claude from the new task you're setting for it and fill its context window with (now irrelevant) details.

If you've finished up a task with Claude and are ready to start on something new, either:

Open a new terminal window and start a fresh Claude Code session
Use the slash command /clear inside of your current Claude Code session. /clear clears the previous conversation history and frees up context so you can begin again.

Targeted conversations help keep Claude on task and less prone to wander or touch files not related to what it's currently working on.

If you like GUIs, try the Claude Code VS Code extension

If you're a diehard IDE fan like me, you may really dig the Claude Code native extension you can install inside of VS Code (or your IDE of choice). The extension got a big upgrade since the release of Sonnet 4.5 and Claude Code v2 a few weeks back, and it has many of the slash command features in Claude Code built in as drop downs and selection menus. Plus, if you've already got the relevant code open in one window of your IDE and Claude open in another, it's pretty easy to direct Claude to where you want it to focus on in the codebase.

</figure>

Personally, I've gotten so used to working with Claude Code in the terminal that I probably won't switch over to this workflow anytime soon, but if you're newer to using Claude and you like the experience of having your AI agent and your editor all in one, it's definitely worth trying out.

Claude does better with code examples to guide it

Another thing I've noticed about working with Claude Code is that it tends to do better when it has already existing coding conventions within a repo to look to for guidance. This goes beyond what you'd typically include in a CLAUDE.md file though.

For instance, if you have a particular way of handling errors, logging requests, or writing tests, pointing Claude towards these files and asking it to write code that's similar for some new feature you're implementing usually works out pretty well. Don't expect Claude to be aware of helper functions or to recognize when it's writing redundant code for multiple functions - you'll still have to point that out and tell Claude to DRY it up or use the helper function instead of writing a new one, but if there is code that Claude can look to for preferred code syntax or styling, it will help.

On a greenfield project where everything is new, it will be more up to you to make good architectural decisions about your codebase up front and establish code patterns that you like, then to lean on Claude for designing a robust, scalable, maintainable codebase.

Use Claude's planning capabilities

Just like a typical developer, Claude's final code tends to be better when it has planned ahead.

This advice really goes hand in hand with the above tips about giving Claude more context so it can be more effective, but beyond telling Claude to look at particular files, read its CLAUDE.md file, and visit those URLs, there's specific commands and techniques you can use to nudge Claude in the right direction and help it stay on track (and hopefully result in less rewrites when reviewing Claude's work).

Engage Claude's extended thinking modes

I only learned about this trick relatively recently, but Claude actually has what it calls "extended thinking mode", which gives Claude additional computation time to evaluate alternatives more thoroughly (i.e. think through a problem and solution more carefully).

To trigger these increasing levels of thinking, you'll instruct Claude Code to:

"Think"
"Think hard"
"Think harder"
Or "Ultrathink" (the name of this thinking level cracks me up)

<figcaption>Claude's different thinking modes</figcaption>

</figure>

Frequently, I'm less than impressed with Claude's quickest responses when I tell it about a bug I'm encountering while testing a new feature it's helped implement, because Claude tends towards the simplest solution that fails to take into account the nuances of this bug (or the impact that the solution might have on other parts of the system that also use the same helper function, reusable component, API endpoint, etc.).

In these instances, usually after Claude's first attempt to fix it has failed, I'll tell it the same bug is still there and to "think" or "think hard" about how to solve this. This will trigger Claude to spend more time on the solution: it will read and search the repo for files related to the bug (if you've provided a screenshot or pointed to a specific file that might be the culprit), maybe run a bash command or two, then eventually provide a better solution to solve the problem in a more thoughtful manner.

If you want to see what Claude is thinking to itself, in the terminal, you can expand a condensed thinking block with CTRL + O. Here's an example of the kind of thoughts you might see when you open a detailed transcript of what Claude Code is thinking as it's going about its tasks.

<figcaption>The kinds of detailed notes you'll see when you expand one of Claude Code's condensed thinking text blocks</figcaption>

</figure>

NOTE: I don't believe I've ever invoked Claude's Ultrathink thinking mode, but I've had good luck with its lower levels of thinking. If you've used Ultrathink I'd love to know if you thought its output was worth the extra tokens and processing time.

Tell Claude to make a plan

One tip that helped immensely with Claude, is telling it to formulate a plan at the start of a fresh session or new feature.

Typically, I'll start a new session, tell Claude what I'm trying to build a new feature for, and point it towards any existing relevant code, docs already written about the topic, or any other helpful info I can think of, and then I'll ask Claude to think about how it would tackle this feature and make a plan.

This serves two purposes:

It makes Claude use one of its extended thinking modes, which almost always guarantees a better end-result.
Claude comes up with a discrete set of tasks or todos to accomplish this that I can review ahead of time and approve (or tell it to revise) before it starts actually touching code.

Once I'm satisfied with the plan, I'll tell Claude to write the plan as a Markdown file inside the project so that it can come back to the plan for reference whenever it needs to (like say, if its current context window fills up and it needs to compact the conversation before resuming work), and I also have a refresher document for future me if I need to refamiliarize myself with that feature again. Once I have verified the final code looks good, I'll tell Claude to review the plan document again and make sure it did everything agreed upon up front, as well as update any changes that occurred as the feature work progressed.

At first, I thought about tossing these docs after Claude had completed the work, but lately I've been thinking of keeping them in the project in a /docs/ folder that I can look to (or point Claude towards) in the future. It can serve as another way to keep useful project context close at hand that can be referenced only when needed instead of plugging it all into CLAUDE.md files that Claude will see every time a new session is begun.

These docs are also helpful when submitting PRs in GitHub, because I can reference them for which files were changed, what work was done, and so on, easily.

NOTE: Anthropic introduced a specific Claude Code Plan Mode back in June of 2025 for this very thing. Plan Mode is a special operating mode that allows Claude Code to research, analyze, and create implementation plans without making any actual code changes. To activate it manually in a Claude Code session press Shift + Tab twice.

<figure>

</figure>

I've used it a few times now, and haven't seen a tremendous difference between Plan Mode and when I just tell Claude to plan, but I'd like to know how you think it stacks up if you've tested the two modes as well.

Help Claude help itself with MCP tools

Similar to how devs are constantly opening browser tabs to research, test, read docs, and figure things out, you can give Claude similar powers to help itself through the use of model context protocol (MCP) tools.

MCP is an open-source standard for AI-tool integrations, which allows Claude Code (and other AI agents like Cursor) to connect to hundreds of external tools and data sources so that it can be more useful and effective while doing its work.

There's truly no shortage of MCP servers out there already (many of which are for specific SaaS tools and services), but a couple of more generic MCP servers I've gotten benefit from so far are:

context7 MCP - This MCP provides up-to-date, version-specific documentation and code examples for LLMs and AI code editors. This means that if you're using the latest version of Svelte, but Claude's training data is with the previous version, you can tell Claude to "use context7" to look up the latest docs and use those to guide it.
Playwright MCP - This MCP provides the browser automation capabilities of Playwright so LLMs can open their own web pages, take snapshots, and examine browser output the same way a developer would. This reduces the need to copy paste screenshots of browsers into Claude Code manually.
Chrome DevTools MCP - This MCP lets Claude control and inspect a live Chrome browser. It goes a few steps beyond Playwright by giving Claude access to the DevTools console, network tab, and even things like Lighthouse for perf testing.

Adding any of these MCP servers to Claude's arsenal is easy to do. They all have documentation in their GitHub README.md files showing how to install them in all of the popular AI coding agents, and it's usually something as simple as running a command like this inside of an open Claude session:

claude mcp add playwright npx @playwright/mcp@latest

Once the server is installed, you may need to restart the terminal and resume the session for it to take effect, but going forward, you'll be able to use the slash command /mcp and see all the currently active MCP servers that Claude has access to.

</figure>

Then, when you want Claude Code to use any of these MCP servers, you'll simply instruct it to "use context7 to look up docs for XYZ" or "use Playwright to check what the browser is rendering to the user". You're helping Claude Code to help itself.

NOTE: The first time Claude uses Playwright to pop open a new browser and navigate to some page of a locally running application on its own is a pretty surreal experience. It was for me anyway.

Lean on Claude's expertise in unfamiliar territory

Although Claude's not the best at starting a project from ground zero, it can be very useful for updating files, explaining code, or using programming languages you may be less familiar with because it has been well trained on a large variety of programming languages.

Pair program with Claude to learn while you go

Claude can be helpful as a guide for codebases you're new to, or even just parts of the code you haven't touched before but need to work on now.

For instance, recently I had to update the Terraform infrastructure files in a project I work on, but Terraform is not something I'm well versed in. Other folks on my team are the devops go-to people and I'm happy to let them handle it, but in this case, the task fell to me. I told Claude what I needed to accomplish (update these files to point to a new set of URLs in production), and as it helped me update the files I asked questions along the way to make sure I was getting a better understanding of how the Terraform setup works, and which files should or shouldn't be touched to make sure things kept running smoothly in production.

Now, I hope I don't need to use that newfound Terraform knowledge much in the future, but when I encounter unfamiliar code or a programming language I don't know much about, I feel fairly confident with Claude Code by my side that we can figure it out together.

Let Claude write your [fill-in-the-blank] scripts for you

Shell scripting can be a powerful tool for developers, but it's also one of those syntaxes that, if you don't use it for a while, quickly starts to slip from working memory (at least for me). So in the case when I need a one-off bash script, a regex function, a new GitHub Action workflow, or [fill-in-the-thing-you-always-forget-the-syntax-for-here], Claude can really come in handy.

The trick here though (and I'll go in-depth more on this in the next section), is that you need to be able to quickly test and verify that whatever Claude has given you actually works.

Run that shell script, test that regex, watch that GH Action workflow progress, and check the output. Because Claude may produce something that looks sensible (especially in a programming language you don't use often or at all), and it will be just as confident in its code that doesn't work as its code that does work until you call it on it. Then you'll get the standard "You're absolutely right!"-response and Claude will try again.

Also, ask Claude to explain how the script works if it's having trouble getting it right. Sometimes, Claude checking its own code helps it find and fix bugs in them as well.

Think outside the box: use Claude for tasks beyond coding

Maybe this isn't news to anyone else, but I've just begun using Claude Code for non-programming related tasks.

Here's a couple of examples off the top of my head:

I had Claude Code draft up a PowerPoint presentation outline (and even a couple of diagrams for the presentation itself courtesy of Mermaid diagrams).
My friend had Claude Code use some family photos and design a nice scrapbook layout to showcase them, that converted to a perfect 8.5" x 11" PDF for printing.

Claude has become synonymous with coding in particular, but it can do a solid job at other things. It can also be quite creative in how it solves a problem since it's limited to the terminal and can't actually generate traditional images. So even if you're tasked with something not directly programming-focused, Claude may be a good sounding board or have some skills that can help you get the job done quicker or in a different way than you may have imagined before.

Don't trust Claude: review and verify

Ok, this is probably the best thing you can do to get the most value out of Claude Code (and have the least amount of refactoring after the fact to bring Claude's code in line with your standards and expectations): review and verify everything Claude does. Claude Code is a useful tool, but it must be watched like a hawk (and its code tested often and thoroughly) to keep it on task and making progress towards the goal you want.

Follow along with Claude's changes in VS Code

In the first version of Claude Code, it would show the code changes it wanted to make to a file inside the terminal itself with the name of the file at the top so devs could see which file it intended to touch. For some devs this may be enough, but for me, I also have VS Code open at the same time to watch the changes being made as Claude goes.

In the current version of Claude Code, there's a setting you can enable in the /config called "Auto-connect to IDE" that lets Claude show the changes it wants to make to the file in VS Code, which is great. It still waits for permission to make the changes or tell it to do something different, but at least this way you can see the other code in the file along with the changes Claude wants to make.

<figcaption>How to enable Claude Code's "Auto Connect to IDE" feature in its config settings</figcaption>

</figure>

When the feature is enabled and the VS Code repo is open on your local machine, Claude will show in its terminal "Opened changes in Visual Studio Code" and pop open the file and display the code changes it wants to make inside the file, very similar to how you see diffs in GitHub PRs. Here's a screenshot of the Claude Code terminal instance and the file in VS Code Claude wants to modify.

</figure>

Following along with the changes in the repo and having my own server running (or automated tests, etc.) to frequently test the changes Claude is making and ensure code is working correctly tends to lead to better outcomes and a shorter feedback cycle.

Do not let Claude YOLO (aka "Allow all edits in a session"). Course correct early and often

This may be an unpopular opinion, but I never let Claude make any code changes without my permission. For best results, you cannot "set it and forget it."

<figcaption>I will never choose option 2: "Yes, allow all edits during this session." Claude needs constant supervision.</figcaption>

</figure>

I have learned through trial and error that allowing Claude to autonomously make decisions without oversight is a recipe for throwing away a bunch of useless code because it took a wrong turn and started going down a bad path and no one stopped it until it was too late.

Yes, it is a bit annoying when Claude asks you for permission to visit a URL you've just given it, or view an image you've pasted into its terminal, but when it makes a code suggestion and you realize it forgot to account for an edge case or it's making an assumption that's false, or it just plain forgot to include something you wanted it to, being able to review the changes, select Option 3 from the dropdown "No, and tell Claude what to do differently", and get it back on the right track is worth the extra time, in my opinion.

It's much easier to course correct earlier on in the process than when the PR is complete and waiting for review, so keep a tight leash on Claude.

Make Claude test its own code changes

One of the best ways I've found to get Claude to catch bugs in the code it's generated is by having it write unit tests for said code. I know that plenty of folks expound on the benefits of Test Driven Development (TDD), but I'm not one of them. I agree with it in theory, but don't actually do it in practice.

So after I've followed Claude's code implementation and done manual testing via the API or the UI (depending on the feature), I'll ask Claude to write unit tests, integration tests, end to end tests - whatever is most appropriate for the situation. Test writing is not something I enjoy doing (all the data setup, the mocks, the spies, the particular testing framework syntax whether you're using Mocha, Jest, Vitest, Cypress, Selenium, etc., etc.), but I know the value it provides, and the speed with which Claude can produce new test files or update existing test files (especially when pointed towards an existing test file that shows the preferred testing libraries and syntax) is great.

By and large, Claude's tests (at least in JavaScript and TypeScript) are good. And they have actually caught bugs in the code that Claude wrote. One example that springs to mind, is a timing function where Claude forgot to add a set of times together for the final value in the actual code, but it did correctly calculate what the final time should be in the test. The unit test revealed the bug and with a little prompting from me to review the actual code implementation, Claude found and fixed its error.

Be warned though, even though Claude will happily run test suites for you all day long, it will sometimes say that failing unit tests aren't due to code changes it made (even though they are) and won't offer to fix them proactively. In this case, I just tell Claude to fix them anyway - I don't really care whose fault Claude thinks it is.

Claude can refactor well

Another thing Claude Code can do pretty well is refactor already existing code, although again, it won't suggest this to you. You must suggest it to Claude.

This might occur when I've had Claude implement a more complex feature: functions get big and unruly and try to do too many things, there's duplicate code for functions that do similar things, or there's just too much code to look at in one file and it needs to be broken down into smaller, easier-to-understand chunks of code.

Many times I've pointed Claude towards a file that's getting out of hand and told it to either simplify the file by breaking down the big functions into smaller ones, search for duplicate code that can be handled with a reusable helper function, or even refactor out code into separate files altogether. Those files can get pretty gnarly at times, and letting Claude sort out what props need to be passed into newly created functions or UI components saves time and makes the code more readable in the future.

Again, test the refactors after they're done to make sure everything still works, but I've had pretty good success with this in TypeScript-based projects.

Have the AI agents review each other's work

Like any true developer, the Claude Code session adding new code to a repo is proud of its own output. Bringing in another, more impartial set of eyes, like that of GitHub Copilot doing a PR review or even another Claude instance reviewing the new code, helps keep Claude more honest.

Typically, I will tag GitHub Copilot as a reviewer when I open a new PR in GitHub, and then I'll take any relevant comments it makes (some comments are irrelevant, so be sure to screen those out using your own good judgement first), and feed them back to Claude Code. Almost always the comments I share with Claude to review and revise end up making the code better and more robust than it was before.

Depending on what GitHub Copilot or Claude Code plans you have, it may make sense for you to have one or the other (or if you really want to go all out, both) review the new code with a critical eye: that's why we have code reviews after all, and if Claude can write code, it can certainly review it as well.

Conclusion

I don't believe, based on what I've seen, that AI agents are going to replace flesh and blood software engineers anytime soon, but they can be pretty helpful in doing a lot of the day-to-day coding we used to have to do by hand.

By far, my favorite tool to help me code faster is Anthropic's Claude Code. There was a learning curve to get over: learning what context to provide it, how to engage its different thinking modes for more complex tasks, and to always manually test new features it's implemented to make sure it's accounted for everything, but overall, I'm happy with the results I get.

I'm sure I'm not using Claude to the extent that some power users may be, but I feel like the things I've picked up about working with Claude Code in the past 6 months or so, have helped me to write good quality code at a faster clip than I would have otherwise (and even improve the DX in existing codebases with nice little, one-off features Claude can whip up in no time).

Thanks for reading, hopefully, if you haven't given Claude Code a try yet, this will inspire you to do so, and if you have, I hope you learned some new helpful tips to get even better results from it.

So what do you think? Did I miss any tips and tricks you've found particularly helpful when using Claude Code? I'd love to hear them.

References & Further Resources

Run Multiple npm Publishing Scripts with Trusted Publishing (OIDC) via GitHub Reusable Workflows

Tue, 18 Nov 2025 00:00:00 GMT

</figure>

Introduction

In the last few months, the npm ecosystem has been plagued by supply chain attacks where OSS maintainers of widely used packages have been phished and versions of these packages containing malicious code have been published to npm for download. Luckily, the broader software community has noticed the compromised libraries early and removed the bad package versions quickly, but it is an ongoing problem.

To try and curb these attacks, GitHub announced changes to strengthen the security around the npm ecosystem in the form of deprecating long-lived publishing tokens stored in GitHub projects and strongly encouraging everyone to begin using trusted publishing (OIDC) workflows, which use temporary, job-specific credentials to publish to npm. These changes were to take effect in November of 2025.

I've previously written about how I created and maintain a JavaScript SDK for my company's API with the help of GitHub Action workflows, and since I first built that repo a couple of years ago, its use has expanded such that it now has two separate npm publishing workflows that run at different times. One workflow publishes new stable versions of the SDK for public use, and one workflow publishes beta versions of the SDK that the dev team can use for internal testing.

When I tried to follow the instructions for setting up trusted publishing in npm, I was dismayed to learn that currently npm only allows for one workflow file to be declared as a trusted publisher.

So what's an npm library to do that has more than one publishing workflow? Trust the production deploy script and relegate the other scripts to using expiring tokens that will have to be manually updated on a regular basis? 🫠

But fear not! All hope is not yet lost.

After turning to the GitHub Discussion thread where people could ask questions and ~~rage~~ express their feelings about the changes, I wrote about my own use case (2 different workflow files that both need to publish to npm regularly) and someone suggested a solution I'd never come across before: GitHub Reusable Workflows.

After a couple of false starts and a little more assistance from the dev community, I had trusted publishing working for both npm publishing workflows and my crisis was averted! 🙌

Today, I'll show you how to use trusted publishing (OIDC) on npm with multiple GitHub Actions workflow scripts thanks to GitHub Actions reusable workflows.

NOTE: Full disclosure, before I turned to other actual devs, I tried to tackle this problem with just the help of Claude Code and it was useless.

It had zero ideas of how to fix this seemingly intractable problem of two publishing scripts with only one trusted publishing input in npm, which just goes to show that as soon as the coding problem gets less common, the assistance of AI agents starts to fail.

In this case, humans FTW!

GitHub Actions reusable workflows

Before we dive into the solution, let's talk about what even are GitHub Actions reusable workflows, because as I said, I'd never even heard of them before now.

GitHub Actions reusable workflows are YAML-formatted files like any other GitHub Actions workflow file, but they allow for the encapsulation of common CI/CD actions that can be called by other workflows, eliminating the need to copy and paste the same workflow logic across different projects, orgs, or even public repos.

For instance, if you deploy multiple web apps to the same cloud environment, a reusable workflow could encapsulate the entire deployment logic, or a reusable workflow could automatically fix code styling issues to ensure consistent code formatting across projects, or (in my case) a reusable workflow can build and publish different versions of an SDK to npm.

As I learned, the key to successfully using a GitHub Actions reusable workflow for trusted publishing with npm is that npm validates the entry point workflow, not the workflow that actually runs the npm publish command.

What I'll cover next, is how to create a single entry point workflow file that will be designated as the Trusted Publisher in npm, and that file will then call the appropriate GH Actions workflow file to do the actual publishing steps.

Create a `publish-switch.yml` file in the `.github/workflows/` folder

So first things first: create a new single entry point workflow file inside of the project repo's .github/workflows/ folder where the other GH Actions publishing workflow files already live.

For convenience, I named this new file publish-switch.yml. This is the file name that will be set in the npm project's Settings tab as the Trusted Publisher, which I'll get to shortly.

Inside of this new publish-switch.yml file, there are two lines of permissions required to activate the OIDC trusted publishing. The permissions for OIDC need to be in this reusable workflows file (and not the files it will call to publish to npm) because publish-switch.yml is the file that npm validates against when doing the actual publishing.

The lines you'll need to include are:

permissions:
  id-token: write # required for OIDC to request ID token to authenticate workflow
  contents: read # switch this to 'write' too if the workflow also needs to push back content to repo (like if you automatically increment an SDK version before publishing)

The full publish-switch.yml may look something like the following. I'll explain what's going on beneath it.

publish-switch.yml

name: Publish switch

on:
  release:
    types: [created]
  push:
    branches:
      - "test-release-**"
    paths:
      - "openapi.yaml"

permissions:
  id-token: write # Required for OIDC
  contents: write # Required for npm beta version updates

jobs:
  publish-prod:
    if: github.event_name == 'release'
    uses: ./.github/workflows/publish-prod-npm.yml
    permissions:
      id-token: write
      contents: read

  publish-beta:
    if: github.event_name == 'push'
    uses: ./.github/workflows/publish-beta-npm.yml
    permissions:
      id-token: write
      contents: write

For the SDK that I publish to npm, there's two ways the different publishing workflows are triggered.

When a new release is created.
- This event triggers a production release to npm after I review the changes to the SDK, bump the package version, and manually create a new release in GitHub.
When a new branch is created whose name begins with test-release-**.
- This event triggers an automated release to npm where a new version of the SDK is built, tagged with a -beta.XX at the end of the current latest version, and published as the latest beta release on npm.

In my reusable workflow file above, I:

Designate event triggers for each type of GH Actions workflow, and specify those same triggers in this file,
Add the required permissions for OIDC trusted publishing,
And then, based on which GitHub event_name triggered the release, determine which workflow publishing file to run (prod publishing or beta publishing).

NOTE: I've omitted the particulars of the actual GitHub Actions publishing files themselves, because they're outside the scope of this article, but if you'd like to see them in their entirety, the production publishing workflow file is here, and the beta publishing workflow file is here.

Configure the npm repo's Trusted Publisher to point towards the `publish-switch.yml` file

Now that the GH Actions reusable workflow file is created, it must be set as the Trusted Publisher file inside of the npm repo's settings.

To do that:

Navigate to the npm project's "Settings" tab (I believe you'll need at least Maintainer level access to the repo to be able to get there),
Select the button for your publisher (at the time of writing this only GitHub and GitLab are available),
Configure the following fields:

Organization or user: GitHub username or organization name
Repository: Repository name
Workflow filename: The filename of the GH Action workflow inside the repo doing the deployment to npm. (In my case it is publish-switch.yml.)

Then click the Save Changes button, and it's done.

</figure>

Voilà! OIDC trusted publishing to npm

All that's left to do at this point is check that the publish-switch.yml workflow is working correctly.

<figcaption>Here's a screenshot in my SDK repo that shows both prod and beta publishes to npm are relying on the Publish Switch workflow and succeeding, back to back.</figcaption>

</figure>

Test npm beta publishing workflow

For me, I tested the npm beta publishing workflow by creating a new test branch in my repo with the name test-release-oidc-publish, and watched the publish-switch.yml file correctly select the publish-beta-npm.yml workflow file to run.

To verify it correctly deployed, I checked the GH Actions workflow logs, and at the bottom of the "Publish Beta Version to npm" step, I can see the text, "npm notice publish Signed provenance statement with source and build information from GitHub Actions".

</figure>

Test npm production publishing workflow

Then, to test the npm production publishing workflow, I created a new release of the SDK code, and once more, watched the publish-switch.yml file correctly identify that this event should run the publish-prod-npm.yml workflow file.

Once again, I checked the GH Actions workflow logs, saw the successful publish message at the bottom of the "Publish to npm" step, "npm notice publish Signed provenance statement with source and build information from GitHub Actions."

</figure>

Success!

NOTE: For one more gut check, I could also visit the npm website and make sure the new beta or production versions are available for download there.

Conclusion

The democratization of coding over the last couple of years has made it much easier for people without backgrounds in computer science to bring their ideas to life online, but it's also made it easier for bad actors to add hard-to-detect malware in popular, previously innocuous software libraries with hundreds of thousands or millions of downloads per week.

GitHub stepped up by quickly introducing new, more secure ways of controlling JavaScript libraries published to the npm package registry and swiftly deprecating older methods more prone to potential hacking. And while this is a good move in theory, since it's been rolled out so quickly, it's left many open source maintainers in a tough spot if they manage many npm packages or have multiple, legitimate publishing scripts they use to publish to npm, because currently projects can only designate one script as a project's Trusted Publishing script.

This one script issue was the problem I encountered with the npm package I maintain. There's two separate workflows with very different steps depending on whether I'm publishing a new production version of the library for public use, or a beta version of the library for the dev team to test against as they build new features.

I thought I would be stuck manually refreshing deprecated publishing tokens regularly to keep both workflows going, but lo and behold, the smart developers in the GitHub community clued me in to a feature I wasn't aware of in GitHub: GitHub Actions reusable workflows.

Through the power of reusable workflows, multiple GitHub Actions workflow scripts can be triggered to handle the different types of package publishing, while the reusable workflow file carries the necessary permissions to satisfy npm's trusted publishing (OIDC) requirements. Just what I needed.

Check back in a few weeks — I’ll be writing more about JavaScript, React, AI, or something else related to web development.

Thanks for reading, GitHub Action reusable workflows are not well documented as an option for those of us who have multiple GitHub Action workflow publishing files to publish to npm, but I hope this solution may work for you like it worked for me.

References & Further Resources

Reviewing the BenQ RD Series Monitor for Developers

Mon, 08 Sep 2025 00:00:00 GMT

</figure>

Introduction

Like many developers, I began working from home full time when the pandemic hit and the office I commuted to every day was forced to close. My initial WFH setup was at the dining room table with my 16" MacBook Pro, a leftover 18" Dell monitor as my second screen, and an old Microsoft wireless keyboard.

<figcaption>v0 of the work from setup. It came from humble beginnings.</figcaption>

</figure>

As the weeks turned into months and it became apparent no one would be going back to an office anytime soon, I began to slowly but steadily build a solid work from home station: laptop stand to raise my laptop to eye level, adjustable desk, extra office chair that was stored in the garage.

Then mid-pandemic, I joined the fully remote tech startup I work for now, Blues, and it was game on: upgraded Logitech wireless keyboard, dual 27" HP monitors, upgraded office chair (with aftermarket rollerblade wheels, I might add), Thunderbolt docking station, external microphone, key light, and Elgato mini prompter (for the weekly web dev podcast I started with my friends). For quite a while now I've felt very dialed in at this command station each day. After 5 years of incremental upgrades, you'd hope that would be the case.

<figcaption>The v1 wfh command station I've been perfecting for years now.</figcaption>

</figure>

About two months ago, out of the blue, the folks at tech product company BenQ reached out to me asking if I'd be interested in trying out one of its RD series programming monitors: a product line designed with developers' needs in mind. Having heard good things about BenQ, but never actually trying their products, I said "Sure!", and received a 28", 4K RD280U monitor shipped to my front door about a week later.

I was a little dubious some of the monitor's claims like coding-optimized color modes and auto dimming features for low-light environments were just marketing fluff, but once I tried it out, boy was I wrong.

Today, I'm going to review the BenQ RD280U 28" 3:2 Monitor for Programming, and tell you how it's improved my day-to-day coding workflow in ways I never anticipated.

Initial assembly and setup of the RD280U was a breeze

The RD280U ships in a what seems like large box, even for a 28" monitor, but all the pieces are well organized and protected, and my assembly of the monitor was done within minutes following the detailed illustrations in the Quick Start guide included in the box.

The box is quite large even though it's not actually that heavy. Based on its bulk, I would recommend you find a second person to help you get it inside and near to where you plan to assemble and use it, if possible.

The screen itself can tilt between -5˚ - 20˚ up and down, swivel right and left 15˚ , and adjust the height on its stand by 4.3" (110mm).

It comes packaged with HDMI, USB-C, and USB-B cords for connecting to your laptop, and also has a display port as well, if that's your preferred connection method. One pleasant surprise for me was the lack of an external power pack that I'd have to secure somewhere behind my desk. Instead there's simply a power cord that runs from the monitor to an outlet.

Additionally, there's a flexible arm version of the monitor (the RD280UA) you can choose instead of the stand the RD280U comes with if you want to clamp it to your desk instead.

Noteworthy RD280U features for devs

Ok, let's talk about the things that developers care most about: the monitor's features! The RD280U has a lot of them.

"Coding Mode"

The BenQ RD series monitors have a feature called "Coding Mode", which is a series of specialized display settings aimed at providing the best possible text clarity for programmers. Coding Mode adjusts settings to sharpen text and reduce eye strain, and has both Dark and Light themes depending on devs' user preferences.

I'm Team Dark Mode all day, by the way.

Below are more specifics on how Coding Mode enhances a typical coding session.

Higher text clarity

This RD monitor has a max resolution of 4K+ UHD, which means four times more pixels and detail than a 1080p display. What this translates to in practice, is less eye strain when looking at code for hours on end. Coding Mode takes this a step further by fine-tuning the contrast, brightness, sharpness, saturation, and gamma settings levels for text, enhancing the overall sharpness and readability of code, even during long working sessions.

Personally, I can see a difference when I compare two IDE windows side-by-side on my existing HP monitor vs the new BenQ monitor: the BenQ monitor does make the code sharper and easier to read.

Anti-reflection screen properties

Until recently (when I was traveling for work and working from coffee shops and hotel cafes), I didn't think too much about screen glare and light reflection. It must just be where my office is set up at home, but it wasn't really an issue for me.

Working remotely, however, screen glare was an issue I had to contend with and the BenQ Nano Matte Panel technology that actively minimizes contrast glare on the screen would have been most welcome.

If you have lights set up behind you that cause reflections on your monitor or even just windows that let the sunlight in during a typical day, you'll probably really appreciate this feature as well.

The Function Bar

An addition the RD monitors include that is very geared towards developers is the addition of a Function Bar at the bottom center of the monitor.

It has a "Coding HotKey" on the front of the bar that looks like a closing HTML tag </>, and when touched, it provides quick access to features like coding modes, screen brightness and contrast, and a customizable "Function Key" button on the bottom righthand side of the bar that can be set to different options you might want quick access to like Color Mode or input source.

<figcaption>Here's a shot of the settings the Coding HotKey displays when tapped on the front of the Function Bar.</figcaption>

</figure>

In addition to the Function Key on the far right of the Function Bar, there's also a power button for the monitor and a center button that displays all the RD monitor's settings (there's a lot of them). The settings are easy to navigate through as the buttons allow for left/right toggling through menus and the naming is also pretty intuitive.

<figcaption>The full settings panel is accessed by pressing the center button on the bottom of the Function Bar.</figcaption>

</figure>

The bar also has some LED indicators on the front next to the Coding HotKey to show which settings are currently on.

It has:

Night Hours Protection indicator (an owl head), which adjusts the screen to minimum brightness for coding in low light or at night,
Brightness Intelligence Gen2 (a polygon-esque lightbulb), that auto detects ambient light and adjusts the screen's brightness accordingly,
And Low Blue Light Plus (a lightbulb), which reduces blue light while preserving color quality.

For me, after toggling through my screen's settings on initial setup and ensuring my Color Mode theme was set to "Coding - Dark Theme", I haven't needed to use the Function Bar or Coding HotKey much since.

I don't have multiple different computers I want to use this monitor with and switch between (although it supports that!), nor do I feel the need to change my Color Mode theme frequently, but for some devs this may be of greater value.

MoonHalo backlight

Another interesting feature I hadn't consciously considered but really like, is the backlight "MoonHalo" built into the RD280U.

It's essentially a ring light built into the back of the monitor that can be set to turn on and provide ambient light while working. It can also auto dim to ultra low brightness with color-balance based on how dark your work surroundings are, which is influenced by the sensor in the Function Bar mentioned above that automatically detects and adjusts the display brightness based on the ambient brightness throughout the day.

The auto dimming feature, I appreciate a lot. It makes me feel less like I'm working in a cave at my desk after dark..

<figcaption>Here's an angled shot of what the MoonHalo looks like on the back of the RD280U monitor. At this time of day, it's at a low brightness level.</figcaption>

</figure>

While the MoonHalo certainly doesn't take the place of a key light if you do any video recording or streaming, it is nice for providing some extra ambient light and offering adjustable color temperature without needing a separate lamp, and doesn't do the "blinding white screen at night scenario" when the monitor screen is set for daytime brightness conditions by default.

Additional monitor features

Those are some of the technologies and thoughtful additions that the RD series monitors offer specifically for programmers via Coding Mode, but there's some additional things worth pointing out in this review as well, that may just sell you on the BenQ RD monitor otherwise.

Display Pilot 2 software

The BenQ line of monitors comes with its own software control app called Display Pilot 2, to let users efficiently control their display settings on their computer.

The app offers a software interface for adjusting many of the settings I outlined above: audio levels, color mode, screen brightness, MoonHalo, switching screen orientation, and so on.

</figure>

Honestly, until I was writing this review, I hadn't even bothered to download the Display Pilot 2 software to my Mac, and while it provides a nice looking interface to control all the bells and whistles from my computer, I don't see myself using it often after I've dialed in my initial settings.

I should also note that while I was messing with the settings via Display Pilot 2, the software crashed a couple of times and caused my BenQ RD monitor to blink on and off more than once, so your mileage may vary when using it.

Multi-source support for seamless source switching

A lot of developers have different rigs for different purposes: a gaming PC, a MacBook work, a personal laptop, or some other combination of devices.

In the past, it's always been kind of a a pain to switch one machine for another to use the same peripherals (keyboard, external monitor, mouse, headphones, etc.).

Well, the BenQ RD monitor can support more than one machine at a time. It has an HDMI port, display port, two USB-C ports, 1 USB-B port, 1 USB-A port, and a headphone jack. It also has a built-in KVM (keyboard, video, mouse) switch to make shifting from one machine to another easy.

Simply connect Computer 1 and Computer 2 to the RD monitor, and both will be identified. Once both are recognized, you can switch the input video source from Computer 1 to Computer 2, and the monitor will also switch the USB signal to computer 2 as well. No muss, no fuss.

I have not tested this feature out myself, as I do not use multiple laptops with my current setup, but I trust the documentation the BenQ website provides about the KVM switch.

My favorite monitor feature (I didn't see this one coming tbh)

By far my favorite feature of this new RD monitor is (and I'm as shocked about this as anyone else considering all the cool tech I've covered up to now): its 3:2 aspect ratio design!

</figure>

Aspect ratio? Really?? That's what you're most jazzed about???

The 3:2 screen aspect ratio

My friends, it is.

I (apparently) am so used to the more standard 16:9 aspect ratio most monitors come with, that after a few days of adjusting to this new, bigger screen where I can see more lines of code or more info in my browser's dev tools without having to scroll, I'm surprised no one really talks about how much of a difference it can make when evaluating external monitors.

According to the BenQ website, this RD280U screen shows 7 more lines of code, but I tell you that for me it is quite a bit more than 7 lines, and I absolutely love it. And all this without me even flipping the monitor into a vertical orientation!

I am very seriously considering getting another RD280U just so I can have this much extra screen real estate for both my external displays at home.

<figcaption>Just look at the amount of code visible on my old HP monitor (right side) vs the amount of code visible on the new RD monitor (left side). It's like night and day.</figcaption>

</figure>

Conclusion

As humans, we adapt quickly to our current conditions until we get so used to it that they become "normal" to us. Another term for this is habituation.

Over the years, I've developed what I consider a very effective and comfortable work from home setup: laptop stand, external monitors, keyboard and mouse, ergonomic desk chair, and even a small teleprompter screen, and I was quite content with it.

Then, BenQ reached out to me with the chance to try one of its RD programming series monitors, and my eyes were opened to features I hadn't even considered I needed as a developer, but made the daily process of writing and reviewing code for hours so much nicer.

A dedicated "Coding Mode" (with both dark and light modes) that enhances text clarity and contrast, anti-reflective screen, a backlight built in to the monitor itself, auto dimming software measuring ambient light, and a 3:2 aspect ratio that dramatically increased my screen real estate.

None of these features were things even on my radar before the BenQ RD280U, but now, I have a hard time imagining working without them.

If you're considering an external monitor to enhance or upgrade your own coding practice, I would strongly encourage you to consider the BenQ RD280U or its other RD series monitors built specifically with developers and programmers in mind. I was skeptical at first that a monitor could really improve my day-to-day workflow, but I was wrong.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading, I hope you’ll check out the BenQ RD line of monitors the next time you need a new monitor. BenQ took the time to understand what improves the developer quality of life, and built a monitor with that really raises the bar with a ton of useful features.

References & Further Resources

Persist Zoom and Bounds in a React Plotly.js Map

Tue, 20 May 2025 00:00:00 GMT

import CodeSandbox from '../../src/components/CodeSandbox.astro'; import YouTube from '../../src/components/YouTube.astro';

</figure>

Introduction

There are a fair number of data visualization, charting, and mapping libraries out there for web developers to choose from. A popular, and flexible, set of such libraries goes by the name of Plotly.

Plotly is used by companies like Intuit and Grafana, which is a pretty good endorsement by itself, so when I needed to build a data visualization dashboard complete with maps and charts for a project at work, I reached for Plotly to help with those parts.

A nice bonus for me is that Plotly not only has a JavaScript version of its library, but it also has a React-specific implementation, and planned to build my web app with the React meta framework Next.js.

The documentation for the JavaScript implementation of Plotly is decent, but the React Plotly docs... not so much. While the React docs mention certain code quirks that are unique to the React way of doing things, there is a distinct lack of code samples to help illustrate how to handle these quirks.

One such quirk I ran into is around controls within a map: typical interactions like zooming in and out, grabbing and scrolling with mouse clicks, and including useful information in map marker tooltips on hover. For me, after I zoomed or scrolled around the map just a bit, when the React app would re-render (as React apps are wont to do) it would reset to its original map coordinates and zoom level like the page had just been loaded.

Not one to be beaten by code, I experimented, read code on GitHub, checked the docs, and eventually found my way to how to persist the map's current state in between re-renders.

In this blog, I'll show you how to avoid your Plotly.js map resetting its zoom level and map bounds during a React re-render, and instead persist the user's last settings so they can keep interacting as if nothing has changed with some simple state handling in the map's parent component.

Here is a working demo in CodeSandbox where you can interact with a map to your heart's content and never have it reset to its original coordinates as you do so.

Install Plotly.js and React-Plotly.js

In order to use Plotly's maps and charts inside of a React-based application, the first thing you'll need to do is install the Plotly.js and React-Plotly.js libraries.

Run this command in the terminal in the root of your project to add them both.

npm install react-plotly.js plotly.js

In other JavaScript projects in the past, I've most often reached for the Mapbox library for my mapping needs, but since I also needed data visualizations with a high degree of customization, instead of having to install two different libraries (one for maps, one for charts), I went with Plotly, which offers both.

Create a Reusable Map Component

Once the Plotly libraries were installed, I created a reusable map component that could simply be passed the data it needs to display.

Mapbox is a mapping library I'm fairly familiar with so I opted for a Plotly Mapbox-style scatter plot map, but I believe a similar solution will work for the other Plotly mapping options as well.

Here is the full code for my <PlotlyMap/> component, I'll explain its contents below.

NOTE: All of my JavaScript-related code is actually written in TypeScript, so if you prefer to use JavaScript, just know that it can be fairly easily adapted to plain JS by removing things like types from the components and variable declarations.

import dynamic from "next/dynamic";
import {
  getMarkerColor,
  findMostRecentWithAqiLevel,
  getAverageValue,
  formatTooltipText,
} from "./utils/mapHelpers";
const Plot = dynamic(() => import("react-plotly.js"), { ssr: false });

const PlotlyMap = ({
  data,
  onBoundsChange,
  mapBounds,
}: {
  data: any;
  onBoundsChange;
  mapBounds: any;
}) => {
  const mapLayout = {
    font: {
      color: "white",
    },
    dragmode: "zoom",
    mapbox: {
      center: mapBounds.center,
      zoom: mapBounds.zoom,
      style: "dark",
    },
    margin: {
      r: 20,
      t: 40,
      b: 20,
      l: 20,
      pad: 0,
    },
    paper_bgcolor: "#191A1A",
  };

  const config = {
    mapboxAccessToken: process.env.NEXT_PUBLIC_MAPBOX_ACCESS_TOKEN,
    scrollZoom: true,
  };

  const preparePlotlyMapData = (data) => {
    const devicePoints = data.map((row) => ({
      deviceId: row.at(-1).device,
      lat: getAverageValue(row, "best_lat"),
      lon: getAverageValue(row, "best_lon"),
      location: row.at(-1).best_location,
      aqiLevel: findMostRecentWithAqiLevel(row),
      timestamp: new Date(row.at(-1).when * 1000).getTime(),
    }));

    const mapData = {
      type: "scattermapbox",
      lat: devicePoints.map((point) => point.lat),
      lon: devicePoints.map((point) => point.lon),
      mode: "markers",
      marker: {
        size: 8,
        color: devicePoints.map((point) => getMarkerColor(point.aqiLevel)),
      },
      text: formatTooltipText(devicePoints),
    };

    return [mapData];
  };

  return (
      <Plot
        data={preparePlotlyMapData(data)}
        layout={mapLayout}
        config={config}
        onRelayout={onBoundsChange}
        style={{ width: "100%", height: "100%" }}
        useResizeHandler
      />
  );
};

export default PlotlyMap;

Let's talk about the props getting passed to the <PlotlyMap/> first.

data - the list of objects with latitude and longitude coordinates to display their positions on the map (plus any other interesting data about the objects that you want to display in a tooltip)
onBoundsChange() - a function to update the mapBounds variables (more on this later)
mapBounds - an object of the map's current latitude and longitude coordinates: map center, map corners, and the current zoom level

After these props is the mapLayout variable that is mostly boilerplate for the Plotly map component setup. The layout sets the background color (paper_bgcolor), font color, and margins for the map container, the map's current center coordinates, zoom, and map style ("dark"), and the ability for a user to click and drag inside the map (dragmode: "zoom").

The next variable is the config variable which includes a Mapbox API token needed to render the Mapbox tile style in the map container (one of the reasons I like Mapbox is because of the variety of nice map styles they have to choose from), and the scrollZoom boolean which allows users to zoom in and out of maps using the scroll wheel on their mouse and/or a two-finger scroll.

Then there is the preparePlotlyMapData() function that takes in the array of objects from the parent component and loops over them to get the object's ID, its location coordinates, and the time that device last reported in. Once the devicePoints are created, the latitude and longitude for each device are fed into the mapData object which renders each point on the map.

NOTE: There are a couple of helper functions inside of the preparePlotlyMapData() function that I did not cover because they're not relevant to this blog post.

The data being displayed in the map is devices that are measuring local air quality, so I'm using the device's recorded air quality to determine the color of the map marker and add relevant air quality data to the marker's tooltip.

Finally, all these variables and functions are used in the <Plot/> component. The data is prepared according to the map's requirements, the mapLayout variable tells the map container how to style itself on render, the config has the Mapbox access token and enables zoom scrolling behavior, the useResizeHandler property enables automatic resizing of Plotly charts when the browser window size changes, and the onRelayout() function accepts the onBoundsChange() function passed by the parent component.

In the next section, I'll talk in detail about what the onBoundsChange() function looks like, but onRelayout() is used to modify the layout of an existing plot (it works for both charts and maps), and allows for dynamic updates to the plot's appearance and configuration without redrawing the entire chart.

onRelayout() is one of the keys to successfully tracking and persisting the state changes in the map across React component re-renders. More details follow in the next section.

Leverage React's useState() to track map state and visible devices in the map container

The other key to ensuring the map's current coordinates, zoom level, and markers persist across component re-renders leverages React useState() hooks in the parent component. There are two pieces of state that need to be tracked: mapBounds and mapDataPoints.

The <PlotlyMap/> component is imported directly into a Next.js app's pages/index.tsx file, so when you see the code below, you'll also see some placeholders for Next boilerplate code in addition to the code directly related to preserving the map's state.

import { useState, useEffect } from "react";
import {
  convertObjectToArray,
  groupEventsByDevice,
} from "@/components/utils/helpers";
import PlotlyMap from "@/components/PlotlyMap";

const inter = Inter({ subsets: ["latin"] });

type HomeProps = {
  data: any;
};

export default function Home({ data }: HomeProps) {
  const rawData = data;

  const [mapBounds, setMapBounds] = useState({
    center: { lon: -98.5795, lat: 39.8283 },
    zoom: 3,
    topLeft: { lat: 0, lon: 0 },
    topRight: { lat: 0, lon: 0 },
    bottomRight: { lat: 0, lon: 0 },
    bottomLeft: { lat: 0, lon: 0 },
  });


  // devices currently visible on the map after a user pans or zooms
  const [mapDataPoints, setMapDataPoints] = useState([]);
  const [visibleDevices, setVisibleDevices] = useState([]);

  {/* function to set current date range in rendered JSX date components */}

  // group all the events together by device ID
  useEffect(() => {
    const groupedEventsByDevice = groupEventsByDevice(rawData);
    const devicesList = convertObjectToArray(groupedEventsByDevice);
    setMapDataPoints(devicesList);

    const filteredVisibleDevices = filterVisibleDevicesWithinBounds(
      devicesList,
      mapBounds
    );
    setVisibleDevices(filteredVisibleDevices);
  }, [rawData]);

  const filterVisibleDevicesWithinBounds = (devices, bounds) => {
    return devices.filter((device) => {
      const { best_lat, best_lon } = device[0];
      return (
        best_lat >= bounds.bottomLeft.lat &&
        best_lat <= bounds.topLeft.lat &&
        best_lon >= bounds.bottomLeft.lon &&
        best_lon <= bounds.bottomRight.lon
      );
    });
  };

  // when map bounds change, filter out devices that are not within the new bounds
  useEffect(() => {
    if (mapBounds.topLeft.lat !== 0 && mapBounds.topLeft.lon !== 0) {
      const filteredVisibleDevices = filterVisibleDevicesWithinBounds(
        mapDataPoints,
        mapBounds
      );
      setVisibleDevices(filteredVisibleDevices);
    }
  }, [mapBounds]);

  const handleMapBoundsChange = (newBounds) => {
    if (newBounds["mapbox._derived"]) {
      setMapBounds({
        center: newBounds["mapbox.center"],
        zoom: newBounds["mapbox.zoom"],
        topLeft: {
          lat: newBounds["mapbox._derived"]["coordinates"][0][1],
          lon: newBounds["mapbox._derived"]["coordinates"][0][0],
        },
        topRight: {
          lat: newBounds["mapbox._derived"]["coordinates"][1][1],
          lon: newBounds["mapbox._derived"]["coordinates"][1][0],
        },
        bottomRight: {
          lat: newBounds["mapbox._derived"]["coordinates"][2][1],
          lon: newBounds["mapbox._derived"]["coordinates"][2][0],
        },
        bottomLeft: {
          lat: newBounds["mapbox._derived"]["coordinates"][3][1],
          lon: newBounds["mapbox._derived"]["coordinates"][3][0],
        },
      });
    }
  };

  return (
    <>
    {/* Next.js head boilerplate */}
      <main>
        <div>
          {/* page tile and date range display components */}
          <div style={{ width: "100%", height: "450px" }}>
            <PlotlyMap
              data={mapDataPoints}
              onBoundsChange={handleMapBoundsChange}
              mapBounds={mapBounds}
            />
          </div>
          <div>
            <h3>
              Number of devices currently shown on map within map bounds:{" "}
              {visibleDevices.length}
            </h3>
          </div>
          <div>
            <h3>Current map bounds:</h3>
            <p>Northernmost latitude: {mapBounds.topLeft.lat.toFixed(4)}</p>
            <p>Southernmost latitude: {mapBounds.bottomLeft.lat.toFixed(4)}</p>
            <p>Westernmost longitude: {mapBounds.bottomLeft.lon.toFixed(4)}</p>
            <p>Easternmost longitude: {mapBounds.bottomRight.lon.toFixed(4)}</p>
          </div>
        </div>
      </main>
    </>
  );
}

export async function getStaticProps() {
  {/* fetch data to display in map here */}

  return {
    props: {
      data,
    },
  };
}

NOTE: I glossed over some of the Next.js code in the component above that's irrelevant to this article like data fetching, CSS styling, and other page elements like date range display components. If you'd like to see the full code, you can check out this Codesandbox demo.

Now let's discuss of the variables being tracked in this <Home/> component.

mapBounds - an object that tracks all of the map component's relevant information: map's center coordinates, zoom level, and coordinates for each map corner
mapDataPoints - all of the objects to be displayed as markers on the map
visibleDevices - once the user's begun interacting with the map, the full list of mapDataPoints is filtered down to just the devices currently visible within the map's visible bounds (I included it here to illustrate how the currently visible objects update and persist on the map even as the component re-renders other parts of the browser with updated data)

Once the data is fetched from the server (in this case, I'm using a CSV of air quality data produced by Blues Airnote devices located all around the world), the <Home/> component uses that rawData variable to group all the air quality events by device ID and then sets that list as the mapDataPoints array that get passed to the map component.

Now that same useEffect() function also calls another function called filterVisibleDevicesWithinBounds(). This function is what sets the visibleDevices state variable that displays a count of the devices visible in the map component based on the map's coordinate bounds underneath the map container in the DOM.

NOTE: There is a small bug with this component in that, before a user has interacted with the map immediately after page load, the map bounds (and thus the number of map markers visible) are unknown to the browser.

As soon as the user does interact, the browser can update all the data points under the map container, but until they do, that data remains unknown. It's annoying, but a small enough inconvenience I'm willing to overlook it for the sake of this post.

Further down in the code, there's a handleMapBoundsChange() function that is attached to the <PlotlyMap/> component's onRelayout() function. When that onRelayout() function is activated by the user interacting with the map, it sends the new map bounds to the handleMapBoundsChange() function which updates the mapBounds state in this component.

When the mapBounds state is updated the second useEffect() function that gets triggered and calculates the amount of devices currently visible in the current map bounds.

The mapDataPoints, mapBounds, and handleMapBoundsChange() function are all passed to the <PlotlyMap/> component and any time the user interacts with the map, the new map bounds and zoom level get passed back to this parent component, and the map state is updated and persisted, even as map coordinates and amount of devices visible on screen are also updated and re-rendered.

Interact with the map component and verify the coordinates and device count persist

If you'd like to see a working demo of how you can interact with a React Plotly.js map full of markers and see other data on the screen update while the map bounds and interactions persist, I've got a CodeSandbox for you to try out.

Open a terminal inside of Codesandbox and type npm run dev to get it started.

In addition to the <PlotlyMap/> component, I also included a couple of Ant Design date picker components to display the entire date range of the data from the CSV (July 16, 2024 to July 18, 2024).

Feel free to zoom on in on different parts of the map, drag and scroll around, and hover over markers for additional info in tooltips. Every time you interact with the map, coordinates and count of devices being displayed on screen should update accordingly.

And if you want to, comment out the onRelayout={onBoundsChange} line of code inside of the <PlotlyMap/> component's JSX, refresh the browser and then try zooming in or navigating around inside the map. You should see now that the count of visible devices and map coordinates displayed under the map don't update despite the user interactions.

Conclusion

Plotly is a great, open source data visualization and map component library, made even more convenient for a JavaScript-heavy web developer like me by the fact that it has a library dedicated specifically to working well with the React JS framework.

Initially when I tried to use React Plotly to render a bunch of objects as markers on a map, shortly after I'd start interacting with the map, when a React re-render would occur I'd be back to square one as if I hadn't touched the map at all.

The trick, I learned, was to keep track of the map's current state in the parent component, and lean on the onRelayout() function inside of the map component itself to persist the map's current view while DOM elements around it changed. Once I started tracking those things, map interactions worked as expected.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading. I hope if you choose to use a Plotly.js map inside of your own React-based project you'll find these tips on persisting the map's current state across component re-renders useful. Enjoy!

References & Further Resources

How to Cross Link Plotly.js Charts in a React App

Wed, 19 Mar 2025 00:00:00 GMT

import YouTube from '../../src/components/YouTube.astro'; import CodeSandbox from '../../src/components/CodeSandbox.astro';

</figure>

Introduction

Plotly is a powerful, highly customizable set of open source, browser-based graphing libraries for JavaScript and Python. It's used by the likes of companies as big as Intuit and Grafana, so when I needed to build a data visualization heavy dashboard for a project at work, I looked to Plotly for help with the charts, graphs, and maps the app required.

Especially appealing to me is the fact that not only does Plotly have a JavaScript version of its library, but it also has a React-specific version, which is one of the JavaScript frameworks I use most day-to-day.

Unfortunately, although Plotly has implementations in multiple languages, they don't have complete feature parity. The feature I was most interested in was the ability to zoom in on a section of one timeseries-based chart, and have any other related charts on the page automatically show the same time selection - a functionality known as cross linking or cross filtering. The Python version of Plotly has the ability to cross link charts right out of the box, but the JavaScript and React versions do not.

Lucky for me, I'm a web developer, and if something's not already available via a library, I'll figure out a way to build it myself.

This article will demonstrate how you can to make your own cross linked Plotly.js charts inside of a Next.js application by using a shared parent component to manage the date ranges.

Here's what the cross linked charts look like and a link to a live demo in CodeSandbox you can interact with.

Install Plotly.js and React-Plotly.js

To use Plotly data visualizations inside of a React-based application, you'll need to install the Plotly.js and React-Plotly.js libraries first.

Run this command in the terminal in the root of your project to add them both.

npm install react-plotly.js plotly.js

As I said earlier, Plotly is a flexible, well documented, open source data visualization library, and the amount of customization it offers for even very complex charts and graphs makes it a good choice for dashboards that need to display this type of information.

Create a Reusable Chart Component

The first thing to do after installing the Plotly libraries inside of a React-based application is to create a reusable chart component that can display different values passed to it from the parent component.

For me, a Plotly line chart was one of the chart types that made the most sense to have on hand, but this sort of reusable React component structure should translate pretty well to many different Plotly chart types.

Here is the full code for my <PlotlyLineChart/> component, I'll go through each of the pieces in it below.

NOTE: All of my JavaScript-related code is actually written in TypeScript, so if you prefer to use JavaScript, just know that it can be fairly easily adapted to plain JS by removing things like types from the components and variable declarations.

import dynamic from "next/dynamic";
const Plot = dynamic(() => import("react-plotly.js"), { ssr: false });

const PlotlyLineChart = ({
  title,
  data,
  property,
  displayValue,
  startDate,
  endDate,
  setDate,
  maxStartDate,
  maxEndDate,
}: {
  title: string;
  data: any[];
  property: string;
  displayValue?: string;
  startDate: Date;
  endDate: Date;
  setDate: ({ startDate, endDate }: { startDate: Date; endDate: Date }) => void;
  maxStartDate: Date;
  maxEndDate: Date;
}) => {
  const layout = {
    width: 640,
    height: 480,
    title: title,
    xaxis: {
      autorange: false, // determines whether or not the range of the axis is computed in relation to the input data
      range: [startDate, endDate],
      rangeselector: {
        buttons: [
          {
            step: "day" as "day",
            stepmode: "backward" as "backward",
            count: 1,
            label: "1d",
          },
          {
            step: "day" as "day",
            stepmode: "backward" as "backward",
            count: 2,
            label: "2d",
          },
          {
            step: "week" as "week",
            stepmode: "backward" as "backward",
            count: 1,
            label: "1w",
          },
          {
            step: "month" as "month",
            stepmode: "backward" as "backward",
            count: 1,
            label: "1m",
          },
          {
            step: "year" as "year",
            stepmode: "backward" as "backward",
            count: 1,
            label: "1y",
          },
          {
            step: "all" as "all",
          },
        ],
      },
      rangeslider: { range: [maxStartDate, maxEndDate] as [Date, Date] },
      type: "date" as "date",
    },
    yaxis: { title: displayValue },
  };

  const prepareChartData = () => {
    if (data.length === 0) return [];

    // loop through each array of device events and create a trace for each device
    const traces = data.map((device: any) => {
      return {
        type: "scatter",
        mode: "lines+markers",
        name: device[0].device,
        x: device.map((event: { when: number }) => new Date(event.when * 1000)),
        y: device.map(
          (event: { [x: string]: number }) =>
            Math.round(event[property] * 10) / 10
        ),
      };
    });

    return traces;
  };

  const handleRelayout = (event: any) => {
    if (event["xaxis.autorange"]) {
      setDate({ startDate: maxStartDate, endDate: maxEndDate });
    } else {
      let rangeLeft = event["xaxis.range[0]"] || event["xaxis.range"][0];
      let rangeRight = event["xaxis.range[1]"] || event["xaxis.range"][1];
      if (rangeLeft && rangeRight) {
        setDate({
          startDate: rangeLeft,
          endDate: rangeRight,
        });
      }
    }
  };

  return (
    <Plot
      data={prepareChartData()}
      layout={layout}
      onRelayout={handleRelayout}
    />
  );
};

export default PlotlyLineChart;

There is a lot of code here, I know, but it's not actually as complicated as it might seem at first.

First: the many props being passed to the component, which serve to make it reusable in the app for many different line charts.

title - the title text to display above the line chart
data - the list of objects and their values to display inside of the chart
property - the name of the property inside of each object to get the value for and display in the graph (for instance, the property of voltage to show device voltage over time)
displayValue - the title to display on the Y-axis of the graph (the X-axis will always show dates)
startDate - the currently selected beginning of the date range
endDate - the currently selected end date of the date range
setDate - a function to update the current startDate and endDate variables
maxStartDate - the earliest date timestamp based on the data present in the parent component
maxEndDate - the latest date timestamp based on the data present in the parent component

Next is the layout object, which is mostly boilerplate for the Plotly component. The layout takes in some of the props like title, the range for the x-axis (startDate and endDate), the maximum date range of maxStartDate and maxEndDate for the rangeslider (the mini version of the line chart underneath the main chart), and the displayValue string for the y-axis.

The rangeselector code which takes up a fair amount of space is actually just directions for a series of buttons that allow users to quickly choose different time ranges to view in the chart. It can accept values like "month", "year", "day", "hour", "minute", or "second".

The prepareChartData() function is where the array of data passed to the chart is transformed into "traces" that the Plotly chart requires. For my scenario, each item in the array has a device property which is a unique ID, a when property that is a timestamp for the x-axis of the graph in Unix time, and whatever numeric value the property prop passed into the component matches. The appropriate data is pulled from each item and the final list of traces data ends up being organized by device ID.

Finally, the handleRelayout() function helps the chart's parent component knows when a user zooms in or out on a particular section of a chart. It uses the setDate() function prop passed from the parent component to ensure that the parent's view of the currently selected dates stays accurate. This is one of the keys to cross linking multiple charts in the project.

These pieces of set up code: the layout variable, prepareChartData(), and handleRelayout() are all passed to the <Plot/> component, and then it can render a JavaScript version of the appropriate data visualization.

Ok, that's enough time spent on this reusable chart component. Let's move on to tracking the date state next.

Leverage React's useState() to track date selections in the charts

The other key to ensuring that all Plotly charts in a React application are showing the same selected date range relies on React's useState() hook in a shared parent component. For my particular Next.js application, it made sense for me to import my <PlotlyLineChart/> component directly into my pages/index.tsx file, so when you view the code below, you'll see a few placeholders for the Next pages boilerplate code mixed in with the code specific to cross linking the charts.

import { useState, useEffect } from "react";
import dayjs from "dayjs";
import {
  convertObjectToArray,
  groupEventsByDevice,
} from "@/components/utils/helpers";
import PlotlyLineChart from "@/components/PlotlyLineChart";

type HomeProps = {
  data: any;
};

export default function Home({ data }: HomeProps) {
  const rawData = data;

  const [initialDate, setInitialDateRange] = useState({
    startDate: dayjs(),
    endDate: dayjs(),
  });

  const [selectedDate, setSelectedDateRange] = useState({
    startDate: dayjs(),
    endDate: dayjs(),
  });

  const [devices, setDevices] = useState<any>([]);
  const [mostActiveDevices, setMostActiveDevices] = useState<any>([]);

  useEffect(() => {
    if (rawData.length > 0) {
      getInitialDateRange(rawData);
    }
  }, [rawData]);

  function getInitialDateRange(data: any[]) {
    data.sort((a, b) => {
      return Number(new Date(a.when * 1000)) - Number(new Date(b.when * 1000));
    });

    // set initial start date and end date for charts
    const startDate = new Date(data[0].when * 1000);
    const endDate = new Date(data[data.length - 1].when * 1000);
    setInitialDateRange({ startDate: startDate, endDate: endDate });
    setSelectedDateRange({ startDate: startDate, endDate: endDate });
  }

  // group all the events together by device ID
  useEffect(() => {
    const groupedEventsByDevice = groupEventsByDevice(rawData);
    const deviceList = convertObjectToArray(groupedEventsByDevice);

    setDevices(deviceList);
  }, [rawData, selectedDate]);

  // set the 5 most active devices
  useEffect(() => {
    if (devices.length > 0) {
      setMostActiveDevices(
        devices.sort((a, b) => b.length - a.length).slice(0, 5)
      );
    }
  }, [devices]);

  return (
    <>
    {/* Next.js head boilerplate */}
      <main>
        <div>
          {/* page tile and date range display components */}
          <div>
            <PlotlyLineChart
              title={"Most Active Device Temperature"}
              data={mostActiveDevices}
              property={"temp"}
              displayValue="Celsius"
              startDate={selectedDate.startDate}
              endDate={selectedDate.endDate}
              setDate={setSelectedDateRange}
              maxStartDate={initialDate.startDate}
              maxEndDate={initialDate.endDate}
            />
            <PlotlyLineChart
              title={"Most Active Device Voltage"}
              data={mostActiveDevices}
              property={"voltage"}
              displayValue="Voltage"
              startDate={selectedDate.startDate}
              endDate={selectedDate.endDate}
              setDate={setSelectedDateRange}
              maxStartDate={initialDate.startDate}
              maxEndDate={initialDate.endDate}
            />
            <PlotlyLineChart
              title={"Most Active Device RSSI"}
              data={mostActiveDevices}
              property={"rssi"}
              displayValue={""}
              startDate={selectedDate.startDate}
              endDate={selectedDate.endDate}
              setDate={setSelectedDateRange}
              maxStartDate={initialDate.startDate}
              maxEndDate={initialDate.endDate}
            />
          </div>
        </div>
      </main>
    </>
  );
}

export async function getStaticProps() {
  {/* fetch data to display in charts here */}

  return {
    props: {
      data,
    },
  };
}

As I mentioned, I put in some placeholders for code that's not really relevant to the cross linking, so bits like data fetching, CSS styling, and some of the other HTML elements like the date range display components are omitted to keep the code cleaner. If you'd like to see the full code, you can check out this CodeSandbox demo.

Let's discuss some of the variables being tracked in this <Home/> component.

initialDate - the maximum starting and ending date range for the charts (determined by the data fetched from the server)
selectedDate - the user selected starting and ending date range being viewed on the charts at any time (on page load these dates are the same as the initialDate object's values)
devices - after data is fetched from the server, it's grouped by device IDs
mostActiveDevices - once the data is grouped by device ID, the devices are sorted by how many data events each one has associated with it, and the top 5 devices are passed to the chart components

Once the data is fetched from the server (in this case, I'm using a CSV of air quality data produced by Blues Airnote devices located all around the world), the <Home/> component uses that data to determine the starting and ending date ranges and sets the initialDate and selectedDate objects via the getInitialDates() function. Each row of data from the CSV has a Unix timestamp associated with it, hence the date manipulations inside the function. These date objects will then get passed to the <PlotlyLineChart/> components further down in the code.

After the first useEffect() that sets the dates, a second useEffect() is called to group all of the data from the CSV by device ID (each row of data from the server has a device ID included in it), and the devices variable is set.

And last, the devices array sorts all the objects in it by how many events each device has, and the top 5 devices are chosen to be set as the mostActiveDevices, which will then be passed to the <PlotlyLineChart/> component as well.

Now all the data needed by the chart components is ready to be supplied to them. The initialDate and selectedDate data gets passed in as startDate/endDate and maxStartDate/maxEndDate. The setSelectedDate() function gets passed to the setDate() function within the component. The mostActiveDevices list is passed in as the data for the chart to render. And the title, property, and displayValue are all strings passed to each chart. For the data I was using, I had values like temp, voltage, and rssi (received signal strength indicator - used to describe wireless signal strength) available to me, so I chose to display those in the line charts.

And that's basically all you need to do cross link your charts in your React application.

The thing to understand about this component is that it controls the state for the dates that gets passed to all the chart components. As the selectedDate is updated via setSelectedDate() by the user interacting with any of the charts, it sends the new selectedDate state down to all the charts to display.

Test out the cross linked charts

If you'd like to see a working demo of these cross linked Plotly charts in action, I've got a CodeSandbox for you to try out.

Open a terminal and type npm run dev to get it started.

In addition to the <PlotlyLineChart/> component, I also included a couple of Ant Design date picker components to display the entire date range of the data from the CSV (July 16, 2024 to July 18, 2024).

Feel free to zoom on in on different parts of different charts or on the range sliders below the charts, or click the range selector buttons above each chart. Every time you interact with one chart, all the other charts should update accordingly. Pretty cool!

Conclusion

When I saw how the Python implementation of Plotly cross linked charts so when a user interacted with one on the page all of them adjusted, I knew I wanted that same functionality in my JavaScript application. Unfortunately, Plotly didn't have feature in the React version of the library, so I had to build it myself.

It turned out to be fairly simple just by having a parent component keep track of the selected date range that was passed to all the chart components, and reacting accordingly when a user updated the date range in one chart by passing that same date range to all the charts.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading. I hope if you want to cross link multiple charts inside of your own dashboard (whether they're Plotly.js charts or another data viz library) you'll consider giving this solution a try and just lifting the date state up one level so it can be passed to all the charts at once. Enjoy!

References & Further Resources

Build a Custom Time Slider Component with Ant Design and Next.js

Fri, 24 Jan 2025 00:00:00 GMT

import YouTube from '../../src/components/YouTube.astro'; import CodeSandbox from '../../src/components/CodeSandbox.astro';

</figure>

Introduction

Part of my job as a software engineer working for a startup involves building proof of concept (POC) web apps quickly, and one of my preferred tech stacks for such a project is a Next.js application in conjunction with the Ant Design component library.

I'm a fan of Ant Design because it's a library I'm familiar with, and it offers a lot of more complicated components (think filterable tables, nested trees, complex forms, date pickers, and so on) that are highly customizable and just work right out of the box.

When I was tasked recently with building a web app that contained a time slider akin to the adjustable sliders you see in interactive weather maps or other time series data applications that would allow a user to adjust the visible time range within a predetermined set of dates, because I couldn't find a suitable, React-based component or package already available, I ended up modifying a basic Ant Design slider component to suit my needs.

My time slider component modifies its tick mark date formatting appropriately based on how big or small the original time span is, it allows users to drag the slider from either end to adjust the currently selected date range, it allows for date ranges set in the future, and it even delineates on the slider between past dates and future dates with different colors and styling.

Today, I'll show you how to build your own custom time slider inside of a Next.js application using the Ant Design <Slider/> component and the Day.js date formatting library.

Here's what the final adjustable time slider component looks like and a link to the live demo site in CodeSandbox.

Install Ant Design and Day.js

The two libraries you'll need to add to a React-based application to build this custom time range slider are the Ant Design library and the Day.js library.

Run this command in the terminal in the root of your project to add them both.

npm install antd dayjs

As I said before, Ant Design is a great, full-featured, very well documented component library I've worked with many times in the past and it makes building complex web apps that contain elements like drawers, cards, cascading selects, and a host of other components much quicker and easier to get going.

Ant Design uses Day.js by default, and Day.js is a lightweight, JavaScript library that parses, validates, manipulates, and displays dates and times with a largely Moment.js-compatible API. If you've ever used Moment to handle dates in a JavaScript application, you'll feel right at home with Day.js, with the added bonus that it's a fraction of the size of Moment.

Ant Design Slider

One particularly useful component that Ant Design offers is a <Slider/> component with a lot of customizable functionality not many other sliders contain.

For my app, I needed a slider that could:

Select a range within the slider's bound (instead of being anchored to one end of the slider or the other),
Allow that range track to be dragged within the slider.
Offer customizable tick marks, tooltips, and range track styles.

Day.js

Working with dates in JavaScript applications is still quite the pain, even today without a supporting date library. The Day.js library is a faster, smaller alternative to Moment.js, an extremely popular library for handling dates and times, with a very similar API and great internationalization support.

To make dates work for my app with the Ant Design slider, I needed a library that could:

Transform human readable dates like "2024-12-01" into Unix millisecond timestamps, because the slider expects numbers (not date formatted objects) to be passed to it in order to work correctly
Format those same dates appropriately for tick marks of varying date spans
Format said date for hoverable tooltips as well.

The Day.js library is capable of all this and more with simple, easy to understand functions and great documentation, and since it's the default library for Ant Design anyway, it make sense to just fall in line with what Ant D knows best.

Create a TimeSlider component

Once those two libraries are installed in the project, it's time to move on to actually creating the <TimeSlider/> component. Below is the starting code for the component. I'll go through all the pieces that comprise it afterwards.

import { Col, Row, Slider } from "antd";
import dayjs, { Dayjs } from "dayjs";

type TimeSliderProps = {
  outerDateRange: { from: Dayjs; upto: Dayjs };
  innerDateRange: { from: Dayjs; upto: Dayjs };
  onChange: (innerDateRange: { from: Dayjs; upto: Dayjs }) => void;
};

const TimeSlider = ({
  outerDateRange,
  innerDateRange,
  onChange,
}: TimeSliderProps) => {
  const min = outerDateRange.from.valueOf();
  const max = outerDateRange.upto.valueOf();
  const values: [number, number] = [
    innerDateRange.from.valueOf(),
    innerDateRange.upto.valueOf(),
  ];

  return (
    <div>
      <Row>
        <Col span={24}>
          <Slider
            range={{ draggableTrack: true }}
            min={min}
            max={max}
            value={values}
            onChange={(value: number[]) => {
              onChange({
                from: dayjs(value[0]),
                upto: dayjs(value[1]),
              });
            }}
          />
        </Col>
      </Row>
    </div>
  );
};

export default TimeSlider;

I created a new file called TimeSlider.tsx where all this code is going to live, and made a basic shell of the component that accepts the props outerDateRange, innerDateRange, and onChange() from the parent component. Later on in this article I'll show you how to set up the required state and functions in the parent component to pass down to this component.

As this is actually a TypeScript file, this component also gets its own TimeSliderProps types defined for each of the props being passed to it. This is where the Day.js library initially comes into play as well, as the type for outerDateRange, innerDateRange, and even the onChange() function expect Day.js dates to be passed to them.

NOTE: All of my JavaScript-related code is actually written in TypeScript, so if you prefer to use JavaScript, just know that it can be fairly easily adapted to plain JS by removing things like types from the components and variable declarations.

Inside of this component we're going to set a bunch of variables required by the Ant <Slider/> component to render properly.

A min and max variable to tell the slider what it's beginning and ending values are.
A values object that is an array of the innerDateRange's from and upto values.

NOTE: For the range slider to work as you'd expect with date objects, the dates must be converted to Unix Epoch time, that is why you'll notice both the outerDateRange and innerDateRange dates are being called with the Day.js library's valueOf() function. This takes the date and turns it into a Unix Timestamp in milliseconds, which can then be passed to the <Slider/> component.

This is the bare minimum of data the Ant Design component will need to actually render onscreen. Then, it's time to create the component with the help of the Ant Design <Slider/> element and pass the necessary details to it.

For a bit of nicer formatting, I also employed Ant Design's <Col/> and <Row/> elements, but those are not required for this component to work. They help the component to lay out in a horizontal fashion and take up as much space as the screen width will allow.

The <Slider/> component itself requires the min and max variables, the values array, the onChange() function, and in order to make it a range slider whose track can be dragged, the range={{draggableTrack: true}} value must be added to the slider's props.

If this component were to receive valid dates right now, it should render, albeit without much to see. Let's keep going.

</figure>

Calculate and style the tick marks to handle different date ranges appropriately

The next step is to calculate the tick marks that will appear along the slider's track showing the range of the slider.

Here's the code that will get added to the TimeSlider.tsx component to make that happen.

import { Col, Row, Slider } from "antd";
import { SliderRangeProps } from "antd/lib/slider";
import dayjs, { Dayjs } from "dayjs";

type TimeSliderProps = {
  outerDateRange: { from: Dayjs; upto: Dayjs };
  innerDateRange: { from: Dayjs; upto: Dayjs };
  onChange: (innerDateRange: { from: Dayjs; upto: Dayjs }) => void;
};

const TimeSlider = ({
  outerDateRange,
  innerDateRange,
  onChange,
}: TimeSliderProps) => {
  const min = outerDateRange.from.valueOf();
  const max = outerDateRange.upto.valueOf();
  const tickCount = 15;
  const tickInterval = Math.floor((max - min) / tickCount);
  const values: [number, number] = [
    innerDateRange.from.valueOf(),
    innerDateRange.upto.valueOf(),
  ];

   const fmt = (date: number) => {
    const d = dayjs(date);
    const { from, upto } = outerDateRange;
    const range = dayjs(upto).diff(dayjs(from)); // Calculate time difference in milliseconds

    // Pick an appropriate level of granularity for the date range we're viewing
    if (range < 1000 * 60 * 60 * 24) return d.format("hh:mm A"); // range is less than a day, show hours and minutes

    if (range < 1000 * 60 * 60 * 24 * 7) return d.format("M/D, HH:mm"); // range is less than a week, show day of the week

    if (range < 1000 * 60 * 60 * 24 * 30) return d.format("M/D"); // range is less than a month, show month / day

    if (range < 1000 * 60 * 60 * 24 * 365) return d.format("MMM D"); // range is less than a year, show month and day

    if (range < 1000 * 60 * 60 * 24 * 365 * 10) return d.format("MMM YYYY"); // range is more than a year, show year and month

    return d.format("YYYY"); // range is more than 10 years, show year
  };

  const marks: Record<number, { label: string; style?: React.CSSProperties }> =
    {
      [min]: { label: fmt(min) },
      [max]: { label: fmt(max) },
    };

  // add ticks to marks
  for (let i = min + tickInterval; i <= max; i += tickInterval) {
    marks[i] = {
      label: fmt(i),
    };
  }

   return (
    <div>
      <Row>
        <Col span={24}>
          <Slider
            range={{ draggableTrack: true }}
            min={min}
            max={max}
            marks={marks}
            value={values}
            onChange={(value: number[]) => {
              onChange({
                from: dayjs(value[0]),
                upto: dayjs(value[1]),
              });
            }}
          />
        </Col>
      </Row>
    </div>
  );
};

export default TimeSlider;

First a couple of new variables are introduced at the top of the component: tickCount and tickInterval.

tickCount is the number of ticks that should be displayed on the slider.
tickInterval is a value calculated by determining how far away from each other the min and max dates are in time and then dividing that difference by the tickCount number to figure out how to evenly space the ticks along the slider, regardless of what the actual date values are.

Then, I created a function called fmt() which handles the heavy lifting of formatting the dates associated with the tick marks appropriately based on how far apart in time the dates actually are.

   const fmt = (date: number) => {
    const d = dayjs(date);
    const { from, upto } = outerDateRange;
    const range = dayjs(upto).diff(dayjs(from)); // Calculate time difference in milliseconds

    // Pick an appropriate level of granularity for the date range we're viewing
    if (range < 1000 * 60 * 60 * 24) return d.format("hh:mm A"); // range is less than a day, show hours and minutes

    if (range < 1000 * 60 * 60 * 24 * 7) return d.format("M/D, HH:mm"); // range is less than a week, show day of the week

    if (range < 1000 * 60 * 60 * 24 * 30) return d.format("M/D"); // range is less than a month, show month / day

    if (range < 1000 * 60 * 60 * 24 * 365) return d.format("MMM D"); // range is less than a year, show month and day

    if (range < 1000 * 60 * 60 * 24 * 365 * 10) return d.format("MMM YYYY"); // range is more than a year, show year and month

    return d.format("YYYY"); // range is more than 10 years, show year
  };

The fmt() function takes in a date (which will eventually be a tick mark), translates it to a Day.js date (d) that can be easily formatted as needed, figures out the range between between the min and max dates passed into the component from its parent, and then formats the d date according to how wide the total timeline range is.

If range is less than a day, the tick marks will be styled to show hours and minutes, if the range is less than a week, it will show day of the week plus hours and minutes, and so on as the range increases. It takes a little bit of effort to decide when it makes sense to switch from one tick display format to the next, but it ends up working out pretty well in practice. Feel free to adjust your formatting of different date ranges to suit your needs.

Next I created the marks that will get passed to the Ant Design slider component.

These marks are created in two parts.

  // marks for each end of the time slider component
  const marks: Record<number, { label: string; style?: React.CSSProperties }> =
    {
      [min]: { label: fmt(min) },
      [max]: { label: fmt(max) },
    };

  // add ticks to marks
  for (let i = min + tickInterval; i <= max; i += tickInterval) {
    marks[i] = {
      label: fmt(i),
    };
  }

The first half of the code, where const marks is declared, is where the marks for each end of the slider are created. Each end's date is passed to the fmt() function to have it correctly rendered based on the length of the time span.

{
  [min]: { label: fmt(min) },
  [max]: { label: fmt(max) },
};

Then, each tick between the two ends is run through the formatting function as well via this for loop.

for (let i = min + tickInterval; i <= max; i += tickInterval) {
  marks[i] = {
    label: fmt(i),
  };
}

Finally, the resulting marks array is passed to the <Slider/> component, along with all the other values covered earlier.

<Slider
range={{ draggableTrack: true }}
min={min}
max={max}
marks={marks}
value={values}
onChange={(value: number[]) => {
    onChange({
    from: dayjs(value[0]),
    upto: dayjs(value[1]),
    });
}}
/>

</figure>

Format the tooltip

After marks for the timeline, it's time to format the tooltip when a user hovers over either end of the range slider. This is actually one of the easier bits of code within this component as well.

Inside of the component under where all the variables are declared at the top add the following function.

import { SliderRangeProps } from "antd/lib/slider";

const TimeSlider = ({
  outerDateRange,
  innerDateRange,
  onChange,
}: TimeSliderProps) => {
// variables declared up here

 const formatter: NonNullable<SliderRangeProps["tooltip"]>["formatter"] = (
    value: string | number | undefined
  ) => `${value ? dayjs(value).format("MM/DD/YYYY HH:mm A") : ""}`;

This formatter() function takes in the current value of the end of the range slider a user is hovering over, and reformats it to a nicely readable date.

Then call this formatter() function within the <Slider/> component at the bottom of the component file.

<Slider
range={{ draggableTrack: true }}
min={min}
max={max}
marks={marks}
value={values}
onChange={(value: number[]) => {
    onChange({
    from: dayjs(value[0]),
    upto: dayjs(value[1]),
    });
}}
tooltip={{ formatter }}
/>

Voila! Nicely formatted tooltips for the time range slider on hover.

</figure>

Style the track and marks to handle current and future dates differently

Now for the extra credit additions to this time slider component: styling future dates differently within the same component. Sounds impossible? I thought so too, at first, but now I can assure you it's not.

There's two changes I wanted to implement to help users know when they were looking at dates in the past/present versus dates in the future with the time slider:

I wanted to change the color of the track from Ant Design's light blue default to a different color, and
I wanted to change change the color of tick marks in the future, as well.

Lucky for me, Ant Design allows for immense customization of its components, even down to things as specific as track and tick mark styling.

The track styling was more complicated, so let's tackle that one first.

import dayjs, { Dayjs } from "dayjs";

const TimeSlider = ({
  outerDateRange,
  innerDateRange,
  onChange,
}: TimeSliderProps) => {
  // other variables declared at top of component here
  const current = dayjs().valueOf();

  // Calculate track styles based on the current date
  const getTrackStyle = () => {
    if (values[1] <= current) {
      // All within past/present
      return {
        backgroundColor: "#91caff",
      };
    } else if (values[0] >= current) {
      // All in future
      return {
        backgroundColor: "rgb(165, 222, 129)",
      };
    } else {
      // Split between past/present and future
      const pastWidthPercentage =
        ((current - values[0]) / (values[1] - values[0])) * 100;
      return {
        background: `linear-gradient(
            to right,
            #91caff 0%,
            #91caff ${pastWidthPercentage}%,
            rgb(165, 222, 129) ${pastWidthPercentage}%,
            rgb(165, 222, 129) 100%
          )`,
      };
    }
  };


return (
    <div>
      <Row>
        <Col span={24}>
          <Slider
            range={{ draggableTrack: true }}
            min={min}
            max={max}
            marks={marks}
            value={values}
            styles={{
              track: getTrackStyle(),
            }}
            onChange={(value: number[]) => {
              onChange({
                from: dayjs(value[0]),
                upto: dayjs(value[1]),
              });
            }}
            tooltip={{ formatter }}
          />
        </Col>
      </Row>
    </div>
  );
};

export default TimeSlider;

In order to get the track to change color midway through (for when it straddled date ranges both in the past and the future), I needed to declare a new variable called current to keep track of what the actual current date is.

At the top of the component where the other variables are declared, current is added to the mix, and it's simply calling the Day.js library's valueOf() function) to get the current date as a Unix timestamp.

Once the current date is identified, I made a function called getTrackStyle() that compares the values captured by either end of the draggable date range slider and sets the color of the track accordingly.

When both slider values are less than or equal to the current time, the entire track is styled light blue, when both slider values are greater than current time (i.e. the future), the entire track is styled light green. When the range spans both past and future dates, the function creates a gradient effect relying on the pastWidthPercentage() function to determine where to transition the colors and the CSS linear-gradient to create a split color effect.

I'll be honest with you: ChatGPT helped me with this track styling magic. It's been extremely helpful for many complex things I've needed to do with Ant Design components, and I'd recommend working with it if you get stuck.

The getTrackStyle() function was passed to the <Slider/> component inside its styles property as styles={{ track: getTrackStyle() }}.

Then, I styled the tick marks based in the future in a different color as well.

  for (let i = min + tickInterval; i <= max; i += tickInterval) {
    const futureDate = i > current;
    marks[i] = {
      label: fmt(i),
      style: futureDate
        ? {
            color: "rgb(73, 163, 15)",
          }
        : {},
    };
  }

To accomplish this, I modified the for loop that went through all the predetermined ticks and compared each one to the current variable declared above. If any of the marks were in the future, those marks got a little extra styling applied to them in the form of a green font color. If the marks were not in the future, they retained their original font color.

Bonus section: I also wanted a "Now" tick mark to clearly delineate where the time slider track went from blue dates in the past/present to green dates in the future.

  const marks: Record<number, { label: string; style?: React.CSSProperties }> =
    {
      [min]: { label: fmt(min) },
      [current]: {
        label: "Now",
        style: {
          color: "rgb(89, 147, 251)",
          transform: "translate(-50%, -36px)",
          fontWeight: "bold",
        },
      },
      [max]: { label: fmt(max) },
    };

To add an extra tick mark entitled "Now" along my time slider track, I modified the marks variable, adding in the current variable as one more pre-defined tick mark, and giving it a label of "Now" and a little styling to stand apart from the rest of the tick marks along the track.

The "Now" tick mark only shows up when the dates passed to the time slider actually cross the current date, but I think it aids in understanding of what's going on onscreen.

With all of that done, the time slider component itself is complete, and it's time to add it to a parent component that will pass it the dates and functions it requires.

</figure>

Import the TimeSlider component into the parent page

Ok, now it's time to give the custom <TimeSlider/> component the props it needs.

In my case, I had other components that also needed to be aware of any changes to the date range for display purposes (I had chart and map child components that needed to be updated as well when selected dates changed), so I had a page within my Next.js app holding the date state variables and onChange() function that were passed to the <TimeSlider/> component.

Inside of the index.tsx file within the pages/ folder, I added the following code:

import { useState } from "react";
import dayjs, { Dayjs } from "dayjs";
import TimeSlider from "@/components/TimeSlider";
import styles from "@/styles/MainContent.module.css";

type HomeProps = {
  startDate: string;
  endDate: string;
};

export default function Home({ startDate, endDate }: HomeProps) {
  const [initialDate, setInitialDateRange] = useState({
    startDate: dayjs(startDate),
    endDate: dayjs(endDate),
  });

  const [selectedDate, setSelectedDateRange] = useState({
    startDate: dayjs(startDate),
    endDate: dayjs(endDate),
  });

  const handleDateSliderChange = (innerDateSpan: { from: Dayjs; upto: Dayjs }) => {
    setSelectedDateRange({
      startDate: innerDateSpan.from,
      endDate: innerDateSpan.upto,
    });
  };

  return (
    <>
      <main className={styles.mainContent}>
        <div className={styles.page}>
          <h2 style={{ textAlign: "center", margin: "20px auto 0 auto" }}>
            Custom Ant Design Next.js Time Slider
          </h2>
          <div className={styles.timeSlider}>
            <TimeSlider
              outerDateRange={{
                from: initialDate.startDate,
                upto: initialDate.endDate,
              }}
              innerDateRange={{
                from: selectedDate.startDate,
                upto: selectedDate.endDate,
              }}
              onChange={handleDateSliderChange}
            />
          </div>
      </main>
    </>
  );
}

export async function getStaticProps() {
  const startDate = dayjs("2024-12-01").utc().format("YYYY-MM-DDTHH:mm:ss[Z]");
  const endDate = dayjs("2025-02-28").utc().format("YYYY-MM-DDTHH:mm:ss[Z]");

  return {
    props: {
      startDate,
      endDate,
    },
  };
}

In the index.tsx page, I created three variables related to the <TimeSlider/> component.

The initialDate and selectedDate React states, which are both objects containing a startDate and endDate property.
The handleDateSliderChange() function, which accepts new startDate and endDate values, and updates the selectedDate state accordingly.

For convenience, I chose to set my start and end date values inside the Next.js getStaticProps() function call which runs before the page renders in the browser, but it's not required, and I set both date state objects equal to the defined dates.

The initialDate values (which end up determining the min and max ends of the time slider won't change again until they are modified in getStaticProps() and the page is refreshed), but the selectedDate values are free to change within those bounds thanks to the handleDateSliderChange() function which will update the state of the selectedDate values whenever the user interacts with the <TimeSlider/> component.

The <TimeSlider/> component was imported into the index.tsx file and the initialDate object was passed as its outerDateRange prop, the selectedDate object was passed as its innerDateRange prop, and the handleDateSliderChange() function was passed as its onChange() prop.

At this point, the component should have everything it needs to work.

Test out the Time Slider component

If you'd like to see a working demo of this <TimeSlider/> component in action, I've got a CodeSandbox link for you to use.

Open a terminal and type npm run dev to get it started.

In addition to the time slider component, I also included a few nice extras: some Ant Design date picker components and buttons with values like "Last month", "Last week", and "Last day" to show other ways of manipulating the selectedDate state controlled by the parent component.

If you'd like to see how the time slider adjusts to larger or smaller date ranges or play around with past and future dates, just adjust the dates inside the getStaticProps() function in the index.tsx file and refresh the browser.

Conclusion

As I was building a React-based proof of concept web app that needed a time slider that users could drag and adjust to zoom in or out on particular pieces of data over time, I couldn't find a ready-made solution for such a need. So I built one myself.

Using the Ant Design component library's highly customizable <Slider/> component as the base and manipulating dates with the help of the Day.js library, I was able to build a flexible, full-featured, and decent looking time slider without a whole lot of extra code on my end.

Then I took it a few steps further making the slider handle future dates and style them differently and even display a little "Now" tick mark when the slider crossed over dates in the past to dates in the future dates. It's a pretty specific use case, I know, but I think there's more possible applications for interesting sliders like this once you start to think beyond the typical date time options.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading. I hope if you need a way to control time series data views within an app and don't want to rely on just date pickers or inputs, you'll consider giving this solution a try. Enjoy!

References & Further Resources

Automatically Publish a Repo as a PyPI Library with GitHub Actions

Mon, 21 Oct 2024 00:00:00 GMT

</figure>

Introduction

Last year I had the opportunity to turn my company's API into a JavaScript SDK by relying on the project's openapi.yaml file and the OpenAPI Generator CLI.

Notehub JS

For the full details of how I did created Notehub JS, I encourage you to read the original blog post I published about it here.

I did this as much for myself as for other developers, because I and a group of my coworkers were building a lot of JavaScript-based web apps to display and interact with the IoT data our company Blues specializes in transporting from a device in the real world to the cloud via cellular. To make it easier to update the JavaScript library as the API it's based on continues to grow and evolve, I set up a bunch of GitHub Actions workflows to do most of the tedious, repetitive tasks for me, and I learned a bunch of useful new things along the way.

This year, one of my other coworkers who works frequently in Python, asked if I could create a Python SDK for the API, and I agreed, feeling pretty confident about most of the steps involved. One new thing I did have to learn was how to build and publish a Python package to the Python Package Index, PyPI.

In this blog, I'll show you how to set up a GitHub Actions workflow to automatically publish a new version of a GitHub project to PyPI when a new release is made - no muss, no fuss, very little manual input required.

Notehub Py

As with the original Notehub JS project, the Notehub Py project's structure is a bit different from your typical repo because it is automatically generated from the Notehub API's openapi.yaml file. The openapi.yaml file follows the OpenAPI specification standards, and can be used with the OpenAPI Generator CLI to build a Python-based SDK to interact with the Notehub API in just a few commands from a terminal.

Here's a simplified view of the Notehub Py repo's folder structure:

.
├── .github/
│   └── workflows/
│       └── GH Action files
├── lib_template/
│   └── python library template files
├── src/
│   ├── notehub_py/
│   │   └── Python-based API and model files
│   ├── docs/
│   │   └── MD documentation
│   ├── test/
│   │   └── unit tests
│   ├── dist/
│   │   └── bundled .tar and .whl binaries for PyPi
│   ├── pyproject.toml
│   ├── requirements.txt
│   └── setup.py
├── openapi.yaml
├── config.json
├── README.md
└── scripts.py

The openapi.yaml file lives at the root level of the project along with its config.json file, a scripts.py file, and a few other other bits and pieces (license, contribution guidelines, code of conduct, etc.), but the real meat of the library lives inside of the src/ subfolder.

What's unique about this subfolder is that it is regenerated each time the openapi.yaml file is updated, and it has all the API endpoints, models, docs, and the dist/ folder for the notehub-py SDK that actually gets published on PyPI. Publishing just a subfolder of a project inside a GitHub repo is a little unusual, but not to worry, it can be done in an automated fashion.

Create a `scripts.py` and `config.json` file to automate generating the library and packaging it for distro to PyPI

To automate the build and publish steps to deploy Notehub Py to the PyPI registry, we need a couple of things: a config.json file and a scripts.py file.

The config.json file is a configuration file of additional properties used by the OpenAPI Generator and its Python library template to define certain variables like package name, package version, GitHub repo URL, etc. Here is what the Notehub Py's config.json file looks like.

config.json

{
  "packageName": "notehub_py",
  "packageUrl": "https://github.com/blues/notehub-py",
  "projectName": "notehub-py",
  "packageVersion": "1.0.2"
}

Every time a new version of the Notehub Py library needs to be published to PyPI, the packageVersion for the project will be updated in this file, then the commands to regenerate the library and its distribution packages (which I'll cover next) gets run.

The scripts.py file is a set of reusable commands to automate the steps of updating this repo based on the latest version of the openapi.yaml, and packaging it up for publishing to PyPI.

scripts.py

import sys
import subprocess
import shutil
import os

def generate_package():
    try:
        subprocess.run([
            "openapi-generator-cli",
            "generate",
            "-g",
            "python",
            "--library",
            "urllib3",
            "-t",
            "lib_template",
            "-o",
            "src",
            "-i",
            "openapi.yaml",
            "-c",
            "config.json"
        ])
    except Exception as e:
        print("Exception when generating package: %s\n" % e)


def build_distro_package():
    try:
        os.chdir("src/")
        # Check if the 'dist/' folder exists
        if os.path.exists("dist"):
            # If it exists, delete it and its contents
            shutil.rmtree("dist")

        # Upgrade the 'build' module
        subprocess.run([
            "python3",
            "-m",
            "pip",
            "install",
            "--upgrade",
            "build"
        ])

        # Generate a new 'dist/' folder  
        subprocess.run([
            "python3",
            "-m",
            "build"
        ])  
    except Exception as e:
        print("Exception when building distro package: %s\n" % e)       


if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: python3 scripts.py [generate_package | build_distro_package]")
        sys.exit(1)
    
    script_to_run = sys.argv[1]
    if script_to_run == "generate_package":
        generate_package()
    elif script_to_run == "build_distro_package":
        build_distro_package()
    else:
        print("Invalid script name. Use one of: generate_package, build_distro_package")
        sys.exit(1)

The first function, generate_package(), uses the subprocess module to run the openapi-generator-cli tool to generate a new version of the Notehub Py library. Let's break down each of the arguments in the command:

"generate": Specifies the action to generate code.
"-g", "python": Indicates that the target language for the generated code is Python.
"--library", "urllib3": Specifies that the generated code should use the urllib3 library for HTTP requests.
"-t", "lib_template": Points to a template directory named lib_template that contains custom templates for the code generation.
"-o", "src": Sets the output directory to src, where the generated code will be placed.
"-i", "openapi.yaml": Specifies the input OpenAPI specification file.
"-c", "config.json": Uses a configuration file named config.json to customize the generation process.

The second function, build_distro_package(), builds the distribution package for the the Notehub Py project that will be deployed to PyPI.

First, the function checks if a dist folder exists inside the src folder where all the Notehub Py API code lives, and deletes the older version of the folder if it does exist to ensure any previous build artifacts are removed before adding a new distro package.

os.chdir("src/")
# Check if the 'dist/' folder exists
if os.path.exists("dist"):
    # If it exists, delete it and its contents
    shutil.rmtree("dist")

Next, the function upgrades the build module to ensure the latest version is installed. This module is essential for generating the distribution packages.

subprocess.run([
    "python3",
    "-m",
    "pip",
    "install",
    "--upgrade",
    "build"
])

Finally, the function generates a new dist directory by invoking the build module to create the distribution package, which includes both source distributions and wheel distributions.

subprocess.run([
    "python3",
    "-m",
    "build"
])

Both of these functions can be invoked from the command line by running python scripts.py generate_package or python scripts.py build_distro_package.

Set up a PyPI account and library to publish to

Now that the Notehub Py config file and scripts to automate the generation and building of the SDK are done, it's time to prepare PyPI to receive the library.

Follow these steps:

Create a PyPI user account if you haven’t done so already.
After logging into PyPI, go to the “Account Settings” page and select the “Publishing” tab.

<figcaption>Get to this screen by clicking the “Account Settings” tab in the user’s dropdown menu, then select “Publishing” under “Your account”.</figcaption>

</figure>

Scroll down to the “Add a new pending publisher” section of the page, and input the details of your project you want to publish to PyPI: publisher platform (GitHub), project name to display on PyPI, project owner, repo name, publishing workflow name (something like publish-pypi.yaml), and click the "Add" button when you're done.
1. Using PyPI’s trusted publishing option utilizes OpenID Connect (OIDC) technology to provide credential-free publishing authority to trusted third party services like GitHub Actions. This allows us to automate the release process without needing to use API tokens or passwords.

</figcaption>

Now, we're ready for the final step in the process: a GitHub Actions workflow to automate publishing the newly generated distribution packages to PyPI.

Write a GitHub Actions workflow to publish to PyPI

Need a refresher on GitHub Actions?

If you want a quick primer on what GitHub Actions are, I recommend you check out a previous article I wrote about them here.

For me, it made the most sense to trigger a GitHub Actions workflow by creating a new release in GitHub to publish the Notehub Py src/ subfolder to PyPI.

GitHub specifically defines a release as a deployable software iteration that you can package and make available for a wider audience to download and use, which is exactly what I want.

Once I'd chosen this as the trigger for my workflow, it was a pretty straightforward set of steps to publish to PyPI.

Here's what the finished publish-pypi.yml file looks like inside of the ./github/workflows/ folder - I'll break it all down below.

publish-pypi.yml

name: Upload Python Package

on:
  release:
    types: [created]

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: 
      name: pypi
      url: https://pypi.org/p/notehub-py
    permissions:
      id-token: write # IMPORTANT: this permission is mandatory for trusted publishing
    defaults:
      run:
        working-directory: ./src
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.x"

      - name: Install dependencies
        run: |
          python3 -m pip install --upgrade pip
          pip install build

      - name: Build package
        run: |
          python3 -m pip install --upgrade build
          python3 -m build

      - name: Publish package to PyPI
        uses: pypa/gh-action-pypi-publish@release/v1
        with:
          packages-dir: ./src/dist/

Each GH Actions workflow file needs a name, so I chose: Upload Python Package. It tells users exactly what this script's purpose is.

As I said earlier, this workflow is triggered whenever a new release is created in GitHub, which is where the following lines take effect.

on:
  release: 
    types: [created]

on is how a workflow is triggered.
release is the event that triggers the workflow.
types: [created] is the activity type for a release event that triggers the workflow. This gives us more fine-grained control of when the workflow should run.

Then the jobs section runs inside of the workflow. This particular script only has one job, deploy, but if there's multiple jobs, they'll run sequentially unless otherwise specified.

The deploy job defines that it runs on the latest version of Ubuntu in runs-on.

The job is configured to operate within an environment named pypi, with a URL that points to the PyPI project page for notehub-py, and the permissions section grants the job the necessary id-token: write permission, which is required for trusted publishing to PyPI. All of this will correspond to the details filled out in PyPI in the previous section setting up the trusted publishing option via OpenID Connect.

 environment: 
      name: pypi
      url: https://pypi.org/p/notehub-py
    permissions:
      id-token: write # this permission is mandatory for trusted publishing

The defaults.run section sets the working directory to ./src for all run steps, ensuring the commands are executed within the source subdirectory of the project.

Finally, we get to the steps.

The steps are as follows:

Check out the code so the workflow can clone the repository's code into the runner with actions/checkout@v4.
Set up a Python environment with version 3.x using actions/setup-python@v5.
Upgrade pip and install the build modules necessary for building the distribution package.
Build the distribution package using python3 -m build.
Publish the built packages to PyPI using the pypa/gh-action-pypi-publish@release/v1 action. The packages-dir parameter specifies the directory containing the distribution files (./src/dist/).

And there you have it: each time a new release is created in the Notehub Py repo, this GitHub Actions workflow will run and deploy the updated code to PyPI.

Conclusion

After publishing my first JavaScript SDK of my company's API on npm last year, I was asked to build a similar SDK in Python and distribute it to PyPI.

I was able to repurpose a lot of the same steps and workflows I used for generating the JavaScript library to generate the Python library, but configuring the deployment to PyPI was a bit different as that package platform recommends using trusted publishing to deploy new package versions.

With just a few Python scripts, a config file, and a little bit of set up on the PyPI site, I was able to deploy the auto-generated subfolder inside of the Notehub Py project to PyPI, helping developers more easily interact with the Notehub API. GitHub Actions allowed me to build the code, package it up for distribution, and publish it to PyPI in no time.

Check back in a few weeks — I’ll be writing more about the useful things I learned while building this project in addition to other topics on JavaScript, React, IoT, or something else related to web development.

Thanks for reading. I hope learning how to deploy a subfolder of a project to PyPI through GitHub Actions workflows comes in handy for you in the future. Enjoy!

References & Further Resources

Notehub Py GitHub repo
notehub-py SDK on PyPI
OpenAPI Generator CLI

Animate an Auto-Scrolling Carousel with Only HTML and CSS

Fri, 23 Aug 2024 00:00:00 GMT

import YouTube from '../../src/components/YouTube.astro';

</figure>

Introduction

As web developers working with powerful computers we don't often stop to consider that while things like memory and stable Internet connections aren't a problem for us, they may be where our apps end up running.

Case in point: building a web app to run on a Raspberry Pi - which could be quite possible for something like a kiosk in a mall, a doctor's office or even an amusement park.

Last year, I was tasked with building a small demo for work to demonstrate how to package up a simple web application and download it to a Raspberry Pi using a cellular-based Blues Notecard.

If you're interested in how to do this, you can read the full details here. But that is outside the scope of this post.

I decided a good demo would be a weather app where a list of cities would be provided, and the app would fetch the weather forecast for each city and display it in an endlessly looping carousel.

In order to keep the app size small (and quicker to download via cellular data to a Raspi), I built it with pure HTML, CSS, and a little vanilla-JavaScript.

For the auto-scrolling feature of each city's forecast I'd normally look for a carousel library to help me, but since I needed the app to stay lightweight I did some research and found that CSS alone could do what I needed.

</figure>

Today, I'll show you how to use CSS keyframes animations along with scroll snap module and a few well-placed CSS classes to make auto-scrolling, CSS-only carousels possible.

Here's what the final auto-scrolling carousel animation looks like.

CSS keyframes and scroll snap

Before we get to the implementation of this auto-scrolling carousel,let's talk a bit about the CSS features that make it possible: CSS keyframes and the CSS scroll snap module.

CSS keyframes

The @keyframes rule controls the intermediate steps in a CSS animation sequence by defining styles for keyframes (or waypoints) along an animation sequence. This gives more control over the intermediate steps of the animation sequence than CSS transitions.

Not only do @keyframes allow for complex animations without the need for JavaScript, but they also offer a straightforward syntax, and reusability across multiple elements (as you'll soon see in my own code examples below).

CSS scroll snap

The CSS scroll snap module provides properties that let you control the panning and scrolling behavior by defining snap positions. Content can be snapped into position as the user scrolls overflowing content within a scroll container, providing paging and scroll positioning.

It has two properties in particular we'll take advantage of today for the carousel.

CSS scroll-snap-type

The CSS scroll-snap-type property sets how strictly snap points are enforced on the scroll container.

It accepts a variety of different values, but the most common ones are:

mandatory - The scroll container must snap to the nearest snap point after scrolling.
proximity - The scroll container will only snap to a point if the scrolling comes to a natural rest near a snap point.
none - No snapping behavior is enforced.

In practice, you might use mandatory when you need precise control over the final position of the scroll (like with carousels or paginated content where each item should be fully visible after scrolling), and you might use proximity when you want to give users more control over the scroll stopping point, which can feel smoother and less jarring.

CSS scroll-snap-align

The other part of this scroll snap equation, scroll-snap-align, specifies a box's snap position within its snap container when using the CSS scroll snap module.

A child element of the scroll snap container can have the scroll-snap-align property set to control how it snaps into the viewport when scrolling stops. Its values include:

start - Aligns the start edge of the item with the start edge of the scroll container.
end - Aligns the end edge of the item with the end edge of the scroll container.
center - Centers the item in the scroll container.
none - No snap alignment will occur.

With that bit of background out of the way, let's get on to the code.

Set up the HTML and CSS

First, the proper HTML must be set up for the CSS auto-scrolling to work correctly.

1. Wrap the HTML elements to be part of the carousel in a parent div.

To begin, wrap all the slides to be scrolled through inside a parent container. This is where the CSS scroll-snap-type property will be applied so that the child elements will be able to interpret the scroll-snap-align property applied to them.

NOTE: In my full code sample, I defined an HTML template to hold all my weather data for each city, and the template.content.cloneNode() function to clone the template for each city in my list and add it inside the weather-forecast-container element

That's outside the scope of what's needed for this article, but if you'd like to see the full code, it's available here.

Here is my HTML file where the parent container and its children are defined.

index.htm

  <div class="weather-forecast-container">
    <div class="weather-template">
      <div class="current-header-wrapper">
        <div class="current-date"></div>
        <h2 class="location">Loading current weather...</h2>
      </div>
      <div class="current-conditions">
        <div>
          <span class="conditions">
            <img class="condition-icon" src="" />
            <span class="condition-text" />
          </span>
        </div>
        <div class="current-temp"></div>
        <div class="details">
          <div class="wind-speed"></div>
          <div class="rainfall"></div>
          <div class="pressure"></div>
        </div>
      </div>
      <div class="forecast">
        <table>
          <tbody class="future-forecast"></tbody>
        </table>
      </div>
      <div class="weather-carousel-snapper"></div>
    </div>
  </div>

As you can see from the HTML above, the weather-forecast-container class wraps the div with the class of weather-template which holds all the weather info for a given city, and contains an empty div at the bottom with a class of weather-carousel-snapper.

2. Position the empty div at the bottom of the weather-template div to be the carousel's snapper element.

Make sure that this element is inside of the weather-template div and not a sibling to it. It will continue to be empty, but it needs to be present.

<div class="weather-carousel-snapper"></div>

3. Add base styling to the parent container and position the weather template and scroll-snap div inside it.

This is where the CSS scroll snap module comes into play.

styles.css

.weather-forecast-container {
  position: absolute;
  top: 0;
  right: 0;
  bottom: 0;
  left: 0;
  display: flex;
  height: clamp(290px, 88vh, 430px);
  max-width: 780px;
  margin: 8px auto;
  scroll-snap-type: x mandatory;
  scroll-behavior: smooth;
  overflow-x: scroll;
}

Inside the weather-forecast-container class, we're going to go beyond the basic styling of the container like positioning and defining its height and width (both set to the width of a small screen attached to the Raspi), and add the first CSS scroll snap properties here.

scroll-snap-type: x mandatory; - this defines a mandatory snapping in the horizontal axis.
scroll-behavior: smooth; - makes the carousel appear to scroll to the next page, instead of just jumping there as it would by default.
overflow-x: scroll; - overflow content is clipped if necessary to fit horizontally inside the element's padding box.

Next up, is the styling for the weather-template div that contains the weather info and the empty weather-carousel-snapper div.

styles.css

.weather-template {
  position: relative;
  flex: 0 0 100%;
  background-size: 780px 100%;
  background-repeat: no-repeat;
  border: 1px solid rgba(0, 0, 0, 0.25);
  border-radius: 4px;
  padding: 0 10px;
  display: flex;
  flex-direction: column;
  justify-content: space-evenly;
}

I've included the stying for the weather-template div as it's the true parent of the weather-carousel-snapper element, but this is just standard CSS styling.

styles.css

.weather-carousel-snapper {
  position: absolute;
  top: 0;
  left: 0;
  width: 100%;
  scroll-snap-align: center;
}

Last but not least, we add the last bit of scroll snap CSS to the empty div with the class of weather-carousel-snapper.

scroll-snap-align: center; - this will align each item to the center of the container.

4. Create the animation keyframes to auto-scroll the forecast slides

After the CSS styling is done, and scroll snap is enabled, it's time to make the @keyframes animations that will actually make the different cities appear to slide through the viewport from one city to the next.

styles.css

@keyframes tonext {
  75% {
    left: 0;
  }
  95% {
    left: 100%;
  }
  98% {
    left: 100%;
  }
  99% {
    left: 0;
  }
}

@keyframes tostart {
  75% {
    left: 0;
  }
  95% {
    left: -300%;
  }
  98% {
    left: -300%;
  }
  99% {
    left: 0;
  }
}

@keyframes snap {
  96% {
    scroll-snap-align: center;
  }
  97% {
    scroll-snap-align: none;
  }
  99% {
    scroll-snap-align: none;
  }
  100% {
    scroll-snap-align: center;
  }
}

In the code above, there are three separate animations defined.

tonext and tostart - manage the horizontal position of each city's weather forecast at different stages of the carousel, creating a sliding effect either to the right or far left, followed by a reset to the starting position (for getting the animation to endlessly loop back to the first city in the list again).
snap - controls the scroll-snap-align property, adjusting the snapping behavior at specific points in the animation timeline. This creates the dynamic snapping behavior that centers up the next city forecast in the carousel.

By combining these three keyframes, we'll have the slick-looking animation in the demo video where the carousel appears to smoothly slide from one forecast to the next.

5. Apply the keyframe animations to the appropriate HTML elements inside the weather-forecast-container parent div.

Now we're going to apply the @keyframes animations defined in the previous step.

styles.css

@media (hover: hover) {
  .weather-carousel-snapper {
    animation-name: tonext, snap;
    animation-timing-function: ease;
    animation-duration: 4s;
    animation-iteration-count: infinite;
  }

  .weather-template:last-child .weather-carousel-snapper {
    animation-name: tostart, snap;
  }
}

@media (hover: hover) - in an effort to ensure that the carousel animations and scroll snap are not applied to devices where hovering isn't possible, such as with most smartphones and tablets, I wrapped these animations inside of the hover media query.

The .weather-carousel-snapper class is given the tonext and snap animations to run simultaneously with animation-name: tonext, snap;, and it will ease over 4 seconds and loop infinitely.

A rule targeting the last child of the .weather-template element with the class of weather-carousel-snapper, applies a different animation combination (tostart and snap), to reset the carousel after the last city's weather forecast has been displayed onscreen to get the carousel back to it's starting position (and first city in the list).

6. And there you have it: a CSS only, auto-scrolling carousel with just HTML and CSS.

The finished product demoed once more in this video.

Conclusion

In certain situations, it's really crucial that web applications be as small as possible. For instance, when that app may be running on a low power or underpowered system (like a Raspberry Pi).

When I needed to build a small demo app that would download over cellular to a Raspi, I decided to do it with plain HTML, CSS, and a bit of vanilla-JS to keep the zip file size as small as possible - no JS frameworks like React or Svelte to help me out.

I wanted to build a weather app that would display the forecast for a list of cities and auto-rotate through the list before returning to the beginning of the list to start over. At first, I thought I might need a carousel library to make this happen, but it turns out CSS scroll snap and keyframe animations can do exactly the same thing, no JS needed. That's pretty darn sweet.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading. I hope if you need a carousel in the future (auto rotating or not) that you'll try implementing it yourself with CSS - I think you'll be pleasantly surprised how little code is needed to make it happen. Enjoy!

References & Further Resources

CSS-Tricks, CSS-Only Carousel article
MDN docs, CSS @keyframes
MDN docs, CSS scroll snap module
MDN docs, CSS scroll-snap-type
MDN docs, CSS scroll-snap-align
GitHub repo, Cellular-Connected Electronic Kiosk

Lean on CSS Clip Path to Make Cool Shapes in the DOM without Images

Wed, 19 Jun 2024 00:00:00 GMT

</figure>

Introduction

Up until a few years ago if you wanted background shapes or sections of a website that were anything besides rectangles you most likely needed a designer to provide you with a static PNG or JPEG image that would be added as required, but CSS has come a long way since then, my friends.

When I was working on a website update that broke up the contents on the page into different colored background sections, alternating between pure white and soft gray colors, the design mock up I had included one section whose bottom edge tilted up and to the right instead of going across the page at a perfect 90 degree angle, as a typical block element does.

Now I could have asked the designer to make a background image to do this for me, but instead I wanted to see if I could do it on my own with the power of CSS. And lo and behold I could, with CSS clip-path.

Interesting shapes and visuals in the DOM are no longer purely the domain of designers, with tools like CSS clip-path, devs have the power to reshape elements and I'll show you how.

CSS clip-path

If you're less familiar with the CSS clip-path property, like me, it creates a clipping region that sets which parts of an element should be shown. Parts that are inside the region are shown, while those outside are hidden.

<figcaption>A demo from the MDN clip-path docs. Different clip-path options provide different views of the hot air balloon and text.</figcaption>

</figure>

The clip-path property can accept a large variety of values:

<clip-source>, which accepts values like url for an SVG element with clipping path defined.
<geometry-box>, which accepts values like margin-box and border-box.
<basic-shape>, which accepts values like circle() and rect().
global-values, which accepts values like inherit and revert.

The <geometry-box> and <basic-shape> values can even be combined together in one clip-path.

/* this CSS combines two different clip path properties */
clip-path: padding-box circle(50px at 0 100px);

This post doesn't go into great detail about all of the properties clip-path can accept and how they can be combined to create quite complex shapes. If you want more information and examples of clip-path in action, I recommend starting with the Mozilla documentation.

One of the <basic-shape> properties clip-path accepts is polygon(), and this ended up being the solution I needed for my tilted background section.

The polygon I needed to recreate with CSS

<figcaption>The gray polygon background I needed to create with CSS.</figcaption>

</figure>

The image above is a screenshot of the gray background section I needed to recreate with CSS clip-path's polygon() property. And the first thing I needed to do was create some HTML elements to apply the CSS to.

polygon() clip-path vs rect() clip-path

You might be wondering why I chose to use the polygon() property instead of the rect() property with clip-path. While the two are similar, polygon() can create more complex polygonal shapes and offers greater versatility for advanced designs by accepting pairs of coordinates to define each vertex of the polygon, whereas rect() can only handle rectangular shapes.

Set up the HTML and CSS

The site I was working on relied on the static site generator Hugo, a Go-based framework. Hugo uses templates to render the site's HTML, so the example code below should look relatively familiar to you if you know HTML.

A note on templates:

If you've ever used JSX components, Node.js with Pug or Handlebars, or Jekyll - Hugo's templates are similar: HTML elements with Go variables and functions sprinkled in with {{ }} to render the correct information wherever the templates are injected.

Here's the code for what I'd nicknamed the "puzzle section" of the page due to the puzzle piece in the foreground of this section. For the purposes and clarity of this article, I've replaced the Go variables injected into the template with the generated HTML.

single.html

  <div class="about-body">
    <!-- more HTML elements up here  -->

    <section class="puzzle-section section">
      <div class="container">
        <div class="row">
          <div class="col-12 col-md-6 col-lg-6">
              <h4 class="mb-3">
                Lorem ipsum dolor
              </h4>
              <p class="mb-5">
                Sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Ipsum dolor sit amet consectetur adipiscing elit pellentesque.
              </p>
              <h4 class="mb-3">
                Duis aute irure dolor in reprehenderit
              </h4>
              <p class="mb-5">
                in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Consectetur adipiscing elit pellentesque habitant morbi tristique senectus et.
              </p>
          </div>
          <div
            class="col-sm-8 offset-sm-2 col-md-6 offset-md-0 col-lg-6 offset-lg-0"
          >
            <img
              class="img-fluid"
              src="/images/about/puzzle-pieces.png"
              alt="Puzzle pieces"
            />
          </div>
        </div>
      </div>
    </section>

     <!-- more HTML elements below  -->
  </div>

This section of code is relatively compact, but it deserves discussion. In addition to the HTML elements, there are quite a few CSS classes which come from the Bootstrap library, one of the original open source CSS frameworks for responsive web designs.

Among the custom classes like about-body, which I used for adding custom styling, there are classes like container, row, col-12 or col-md-6, mb-5, and mb-3.

All of the latter classes are Bootstrap classes, which serve to make the text and image elements onscreen share the width of the page when the viewport is over a certain width (col-md-6), or apply a margin-bottom of a certain amount to the <p> tags (mb-3 or mb-5).

The Bootstrap classes are beside the point for this post though, the class to focus on is the puzzle-section which wraps all the text and puzzle piece image.

This puzzle-section class is where we're going to add the clip-path property to display the light grey background behind the text and image with the slightly tilted, up-and-to-the-right design.

Add the CSS clip-path to shape puzzle-section

As I wasn't quite sure how to style a normal, rectangular <div> into an uneven shape, I started looking for a solution online and found this helpful, interactive clip-path-focused site, CSS clip-path maker.

</figure>

This CSS clip-path maker website is fantastic because it has a whole slew of preset shapes, adjustable image sizes and backgrounds, and the currently displayed image's vertices can be dragged into any arrangement you want. The line at the bottom of the screen shows the exact clip-path CSS values that you can copy/paste into your own project's CSS.

I chose the parallelogram preset shape as my starting point, and then dragged the corners to match the angle of the background section I was trying to recreate from scratch. Once I was satisfied it looked accurate, I copied the CSS line at the bottom of the page to my clipboard.

In my project's SCSS file, I added the copied clip-path CSS in addition to the light grey background-color property and some padding to give the text and puzzle piece images some breathing room on the page.

NOTE: Even though this file shown in the example code is SCSS instead of pure CSS, for this post it shouldn't make a difference here. It should be a direct 1:1 comparison.

about.scss

.about-body {
  // this white sets the white background color for the whole webpage
  background-color: white; 

  .puzzle-section {
    // clip-path code copied from the clip-path maker website
    clip-path: polygon(0 0, 100% 0%, 100% 75%, 0% 100%);
    background-color: light-grey;
    padding: 2rem 0 10rem 0;
  }
}

That little bit of CSS for clip-path was all that was needed to take my perfectly rectangular DOM element and turn it into an imperfect polygon instead. Not too shabby!

Conclusion

CSS is pushing the boundaries of what web developers can do without resorting to images, videos, and custom designed elements all the time. And the satisfaction of figuring out how to do a cool little bit of design on all on your own feels pretty empowering.

A recent example of this was using the CSS clip-path property to create a background box for some text and images that had an uneven bottom edge. With the help of an interactive website dedicated to decoding clip-paths of all shapes and sizes, I was able to make quick work of this slightly skewed polygon.

And let me take a moment to shout out how much I appreciate the folks putting out those little sites or code snippets that solve a very specific problem for another developer - you folks continue to make the Internet a better place.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading. I hope learning to reshape how elements look in the DOM with just the power of CSS helps you as much as it's helped me.

References & Further Resources

MDN docs, CSS clip-path
CSS clip-path generator website

Create a Custom Formatted CSV from Python Data

Fri, 19 Apr 2024 00:00:00 GMT

</figure>

Introduction

In 2023, a friend of mine asked me to write a program to collect the data from an NFT collection on the NFTrade website. He was interested in the following information: all the NFTs currently for sale in the collection, the current price of BNB (the cryptocurrency the NFTs are listed for sale in), the price of each NFT converted to US dollars based on the current price of BNB, and put all that data into a spreadsheet that's easy to manipulate, sort, and filter.

I couldn't use my go-to JavaScript skills to fetch data via HTTP calls from the NFTrade site because there is no public-facing API, so instead, I taught myself to build a small web scraping script to visit the site and "scrape" that data from it.

Python seems to be a very popular language for such projects, so I went with it, and as I worked on it, the requirements got a bit more complex, and I learned a lot of useful things about Python as a result, which I've shared in a series of blog posts. This post will be the last in my series covering this web scraper.

In previous articles I:

And now, with all the data in one place, I needed to make it into a spreadsheet my friend could work with.

The Python csv.DictWriter module makes turning a list of Python objects into a standard CSV file a straightforward task, which I'll demonstrate in this article.

NOTE: I am not normally a Python developer so my code examples may not be the most efficient or elegant Python code ever written, but they get the job done.

Sample Python data

I need to set the stage and show you the shape of the my Python data before I show you the solution I ended up using. As I mentioned in the introduction, the data I wanted for each NFT included the following things:

NFT ID
NFT list price (in BNB)
NFT rarity score (a number randomly assigned to each NFT in the collection)
Current price of BNB in US dollars
Cost of NFT in US dollars
Cost of each rarity point assigned to the NFT in US dollars

What I ended up with is a list of objects (or "dictionaries" in Python parlance) because each object in the list is made up of key value pairs. Here's a sample of what the list of data looks like.

Sample NFT data to be added to CSV file

[
    {'bnb': 352.44, 'cost_per_rs': 28.2, 'id': 4, 'nft_price': '0.8', 'price_usd': 281.95, 'rs': 10},
    {'bnb': 352.44, 'cost_per_rs': 77.54, 'id': 42, 'nft_price': '1.1', 'price_usd': 387.68, 'rs': 5},
    {'bnb': 352.44, 'cost_per_rs': 98.68, 'id': 174, 'nft_price': '1.4', 'price_usd': 493.42, 'rs': 5}, 
    {'bnb': 352.44, 'cost_per_rs': 29.68, 'id': 184, 'nft_price': '1.6', 'price_usd': 563.9, 'rs': 19},
    {'bnb': 352.44, 'cost_per_rs': 46.99, 'id': 256, 'nft_price': '2', 'price_usd': 704.88, 'rs': 15},
    # more NFT data
]

NOTE: If you're interested in seeing how I got this list of data, check out my previous blog post for a more in-depth explanation.

Ok, now that I've established what the data looks like we can move on to how I turned this list into a CSV, complete with a header row of each key in the dictionary object.

The csv.DictWriter module

Unlike core JavaScript, which tends to be pretty bare-bones and makes users install packages for most everything they want to do, the Python standard library is quite robust.

It comes with a csv module that implements classes to read and write tabular data in CSV format out of the box, as the CSV (Comma Separated Values) format is the most common import and export formats for spreadsheets and databases.

The csv module had two packages of particular interest to me:

writer - a writer object responsible for converting the user's data into delimited strings on a given file-like object.
DictWriter - an object like a writer but it maps dictionaries to output rows. This object requires a fieldnames parameter that is a sequence of keys which identifies the order in which values in the dictionary are passed to the writerow() method.

Since my Python data is formatted as a list of dictionaries, as shown in the previous section, it made the most sense for me to use the csv.DictWriter module.

For me, the fieldnames parameter would be composed of the keys of each dictionary: id, bnb, nft_price, etc. and that would guarantee that each dictionary of NFT info passed through the DictWriter object would match up the proper value with its corresponding key. And DictWriter also has two handy public methods I wanted:

writeheader() - a method to write a row with the field names to the writer's file object (i.e. the top row in a CSV that typically has the column names listed for each column of data).
writerows() - the method that writes all the elements in rows to the writer's file object (i.e. the list of NFT objects I wanted added to the CSV file).

DictWriter was the right choice for my situation, so let's get to how I put it into practice to make my file next.

Format the Python data into a CSV and save it locally

To transform the list of Python dictionaries into a CSV using the csv.DictWriter module, I created a new function called download_csv(), and here is what the code looked like in my for_sale_scraper.py file.

def download_csv(self, card_list):
      """Turn card list into CSV file."""
      date_time = datetime.now().strftime("%Y_%m_%d-%I_%M_%p_")
      # add date time to the front of the file name 
      with open(date_time + 'NFTs_For_Sale.csv', 'w', encoding='utf8', newline="") as output_file:
          dict_writer = csv.DictWriter(output_file, fieldnames=['id', 'bnb', 'nft_price', 'price_usd', 'rs', 'cost_per_rs'])
          dict_writer.writeheader()
          dict_writer.writerows(card_list)
      pprint("CSV NFTs For Sale file generated!")

It will probably be easiest to go through this function line by line, so let's start from the top.

The function takes in a card_list argument which is what I want written to each row in the CSV, and the first thing it does is create a variable named date_time.

date_time = datetime.now().strftime("%Y_%m_%d-%I_%M_%p_")

date_time is a formatted date and time string representing the current date in the format of "Year_Month_Day-Hour_Minute_AM/PM". When called, it will generate a string like "2024_04_11-03_30_PM_", which will be tacked on to the beginning of the CSV's file name so it's easy to sort and identify the files after they're generated.

On to the next line of the function:

with open(date_time + 'NFTs_For_Sale.csv', 'w', encoding='utf8', newline="") as output_file:

with open opens a file for writing with the specified filename, which is the newly created date_time variable concatenated with the "NFTs_For_Sale.csv" string. The 'w' mode indicates the file will be opened for writing, the encoding='utf8' parameter specifies the character encoding to be used, and the newline="" ensures the proper newline character is used for writing rows to the CSV file. The whole thing is named output_file for reference later in the function.

Then we come to the DictWriter.

    dict_writer = csv.DictWriter(output_file, fieldnames=['id', 'bnb', 'nft_price', 'price_usd', 'rs', 'cost_per_rs'])
    dict_writer.writeheader()
    dict_writer.writerows(card_list)

It is DictWriter's time to shine, as a new instance of it is created and the output_file is passed to it, along with the fieldnames list which includes all the dictionary keys as the headers for the CSV file. In the following two lines of code, DictWriter writes the header row to the CSV file with the names specified in fieldnames, and writes the list of dictionaries (the card_list passed to this function) to the CSV file. Each dictionary in the list represents a row in the CSV file.

Job complete!

pprint("CSV NFTs For Sale file generated!")

Finally, a success message is printed out at the end of the function indicating the CSV file has been successfully generated, and the file should show up in the repo alongside all the other previously project files.

<figcaption>Note all the generated CSV files, neatly ordered in descending order by their filename dates.</figcaption>

</figure>

And the download_csv() function is called from the main Python function like in the code snippet below.

I've included the other functions called before download_csv so you can see how I divvied up the work this file was doing, but I have not included the details of those functions because it's outside the scope of this post.

for_sale_scraper.py

if __name__ == '__main__':
   scraper = ForSaleNFTScraper();
   cards = scraper.get_cards(max_card_count=200)
   card_data = [] 
   for card in cards:
    info = (scraper.get_nft_data(card))
    card_data.append(info)

   # filter out any extra cards that aren't for sale
   cards_for_sale = scraper.filter_priced_cards(card_data)

   # add rarity scores to all cards in the list by merging them with the id_rs_list
   cards_with_rs = scraper.get_cards_rarity_score(cards_for_sale)

     # add the current bnb price, current usd price of cards and current usd price of each rs point
   cards_with_bnb_rs = scraper.add_pricing_to_cards(cards_with_rs)

   # generate csv file 
   scraper.download_csv(cards_with_bnb_rs)

Now, my friend can generate CSVs of the NFT collection whenever he wants to, or he could even set up a cron job on his machine to run this script on a regular basis and then review the generated files at his convenience.

Conclusion

Building a web scraper to collect data from the NFTrade site was a good exercise for me to learn more about Python, and it was a great opportunity for me to share my learnings in a series of blog posts for anyone else looking to do something similar.

I scraped the data with Selenium Python, narrowed it down to the pieces I wanted, added some extra info like the cost of each NFT in US dollars, and then bundled all data up into a downloadable CSV for easy scanning and sorting.

The extra nice thing about making it into a CSV is that the CSV reading and writing functions are part of the core Python library so I didn't even have to install any third party packages to make it work - it was actually quite straightforward in the end.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading. I hope learning to use Python's csv.DictWriter module to make your own CSVs from list of data comes in handy in your own apps and projects.

References & Further Resources

NFTrade website
Python DictWriter documentation
First blog post about scraping data from a lazy-loading website using Selenium Python
Second blog post about limiting data searches to a particular element on a page instead of the whole page when using XPath
Third blog post about filtering, merging, and updating lists of objects in Python

Filter, Merge, and Update Python Lists Based on Object Attributes

Fri, 23 Feb 2024 00:00:00 GMT

</figure>

Introduction

Last year, I wrote a web scraping program to collect data from one of the NFT collections on the NFTrade site. My friend wanted the following data included in a CSV: all the NFTs currently for sale in the collection, the total price of each NFT in US dollars based on the current market price of the BNB cryptocurrency that the NFT is for sale in, and the price in USD per rarity point (a value randomly assigned to each NFT in the collection).

The NFTrade website does not have a public API so instead of writing a Node.js script to fetch the data via HTTP calls, I built a small site scraping script to go to the website and actually "scrape" the data from it.

Having not written a web scraper before, I chose to write the program in Python, and as I built the scraper, the project requirements got a bit more complex, and I learned a bunch of useful techniques when coding in Python, which I'm sharing in a series of posts.

After choosing the Selenium Python package to use Selenium WebDriver to scrape the data from NFTrade and extract the details from each NFT that I wanted (the NFT's ID and price in BNB), I needed to update my new list of NFT data in several ways:

I needed to filter out any NFTs that weren't currently for sale (some that were scraped off the site weren't actually for sale),
I needed to match all the NFTs for sale with their "rarity scores" (as defined in a separate JSON list) and include those scores along with the rest of the NFT data,
I needed to compute the total cost and cost per rarity point for each NFT in USD based on the current market price of BNB and add those prices to each NFT in the list as well.

I know this sounds quite complicated, but I broke each of these requirements down into separate methods inside my Python script and learned a lot about working with lists in Python along the way.

Today, I'll show you how to filter lists by whether an attribute exists in an object, how to merge two lists of items together based on matching attributes, and even how to add new object properties to the objects within a list in Python.

NOTE: I am not normally a Python developer so my code examples may not be the most efficient or elegant Python code ever written, but they get the job done.

Sample Python Data

Before I dive into the specifics of my list manipulations in Python, let me give you a little background on what the data looks like that I was working with. Here's a small sample of what the list of NFT data looked like before I started mutating it.

Sample NFT data scraped from the NFTrade site

[
    {'id': 6774, 'nft_price': '0.22'},
    {'id': 5710, 'nft_price': '0.16'},
    {'id': 3187, 'nft_price': '0.8'},
    {'id': 6482, 'nft_price': '1.1'},
    {'id': 7689, 'nft_price': '0.5'},
    {'id': 335, 'nft_price': '4'},
    {'id': 7025, 'nft_price': '1.057'},
    {'id': 597, 'nft_price': '5'},
    {'id': 3936, 'nft_price': '3.1'},
    {'id': 2834, 'nft_price': '0.649'},
    {'id': 763, 'nft_price': '1.65'},
    {'id': 7683, 'nft_price': None},
    {'id': 7914, 'nft_price': None}
 ]

As you can see from the output above, the original data I started with was pretty sparse: the ID number for each NFT and the price in BNB (if it existed) were the only pieces of data present in each object from the info scraped off the NFTrade site. I had my work cut out for me to clean this list up and add more useful data to it, so let's move on to how I did so in the next section.

NOTE: If you'd like to see more about how to scrape the browser data and gather just the necessary bits, read my first couple of blog posts here and here.

Filter objects in a list on whether an attribute exists or not

As I mentioned in the introduction, the first thing I needed to do was clean this list up by removing any NFTs that didn't have a price.

Due to how I had to lazily load and scrape the data from the NFTrade website initially, there was a good chance there were a handful of NFTs I gathered up that weren't for sale, and therefore didn't have prices, so I needed to weed them out first.

Technically every NFT in my list had an nft_price attribute, but if there was no price listed in the card's scraped data, the nft_price attribute was assigned None, which proved very useful.

Inside of the __main__ method in my Python script, I'd already scraped the data from the webpage with the get_cards() method, then looped through the NFT data to grab just the bits of relevant data with the get_nft_data() method. Now I wanted to filter down the cards to only include ones listed for sale.

Here's the __main__ method code first:

for_sale_scraper.py

if __name__ == '__main__':
   scraper = ForSaleNFTScraper();
   cards = scraper.get_cards(max_card_count=200)
   card_data = [] 
   for card in cards:
    info = (scraper.get_nft_data(card))
    card_data.append(info)

   # filter out any extra cards that aren't for sale
   cards_for_sale = scraper.filter_priced_cards(card_data)

And here's the method I came up with to filter down to just the NFTs for sale: filter_priced_cards().

def filter_priced_cards(self, card_list):
    """Filter card list to only cards with NFT cost."""

    # filter out any cards in the list that don't have an NFT price equal to None
    cards_for_sale = list(filter(lambda card: card['nft_price'] != None, card_list))
    return cards_for_sale

Let's break down what's happening in the second line of the filter_priced_cards() function.

I used Python's built-in filter() function to iterate over the card_list passed to the function to create a new list named cards_for_sale. The anonymous lambda function inside of filter() takes each card in the card_list and returns True if the nft_price attribute of the card is not None, and False if it is - this is how it filters out all the cards that don't have a price.

The list() function that wraps the filter() converts the result back to a list, because filter() returns a filter object which is an iterator, not a list.

And finally, the new cards_for_sale list is returned.

Merge two lists together by matching object keys

Once the NFTs not for sale have been filtered out, the next step is to add the rarity score to each NFT based on its ID.

For this particular set of NFTs, each NFT had a "rarity score" that had been randomly assigned to it. The rarity scores for each NFT were listed in a separate JSON file in the project and they look like this.

id_rs_score.json

[
  {"id": 1, "rs": 18},
  {"id": 2, "rs": 13},
  {"id": 3, "rs": 14},
  {"id": 4, "rs": 10},
  {"id": 5, "rs": 22},
  {"id": 6, "rs": 13},
  {"id": 7, "rs": 10},
  {"id": 8, "rs": 13},
  {"id": 9, "rs": 13},
  {"id": 10, "rs": 9},
  // more ids and rarity scores ("rs") below
]

I needed to combine my list of cards_for_sale with the rarity scores in the JSON file by matching up the id attribute in each list of objects. For this task, I came up with the following function: get_cards_rarity_score().

def get_cards_rarity_score(self, card_list):
    """Combine rarity scores with card list by ID."""

    # get rs data for each card from json file 
    with open("id_rs_list.json") as file:
        id_rs_list = json.load(file)

    # merge together cards with id_rs_list by their matching ID numbers 
    match_cards_with_rs_list = groupby(sorted(card_list + id_rs_list, key=itemgetter("id")), itemgetter("id"))
    combined_cards = [dict(ChainMap(*g)) for k, g in match_cards_with_rs_list]

    # filter out all the items in the merged list without a "for sale" value 
    filtered_combined_cards = []
    for card in combined_cards:
        if 'nft_price' in card:
            filtered_combined_cards.append(card)

    return filtered_combined_cards

To combine the rarity score with any of the NFT objects contained in the card_list list, the first thing that had to happen was to read the data from the id_rs_list.json file and assign it to a variable.

    # get rs data for each card from json file 
    with open("id_rs_list.json") as file:
        id_rs_list = json.load(file)

Once the JSON list was extracted from the file, the card_list and id_rs_list needed to be merged together based on their matching IDs.

The groupby() function groups elements with the same ID, and then ChainMap() merged the grouped items into Python dictionaries (objects). The result was a list of dictionaries (combined_cards) where each dictionary represented a card with combined information from both lists.

    # merge together cards with id_rs_list by their matching ID numbers 
    match_cards_with_rs_list = groupby(sorted(card_list + id_rs_list, key=itemgetter("id")), itemgetter("id"))
    combined_cards = [dict(ChainMap(*g)) for k, g in match_cards_with_rs_list]

One thing to note: the combined_cards list has every NFT listed from the id_rs_list, not just the ones whose IDs match the IDs in the card_list. So the combined_cards list looks like the data below - but for every item in id_rs_list.

[ 
 {'id': 1, 'rs': 4},
 {'id': 2, 'nft_price': '3', 'rs': 6},
 {'id': 3, 'rs': 22},
 {'id': 4, 'rs': 4},
 {'id': 5, 'rs': 10},
 {'id': 6, 'nft_price': '5', 'rs': 1},
 {'id': 7, 'rs': 1},
 {'id': 8, 'nft_price': '0.1', 'rs': 14},
 {'id': 9, 'nft_price': '1.5', 'rs': 5},
 {'id': 10, 'rs': 1},
 # more IDs and NFT data 
]

Since the combined_cards list had every single NFT in it (not just ones for sale), once more I had to filter the list down so that every item without an "nft_price" was omitted.

    # filter out all the items in the merged list without a "for sale" value 
    filtered_combined_cards = []
    for card in combined_cards:
        if 'nft_price' in card:
            filtered_combined_cards.append(card)

In this case, since there's a (very likely) chance the NFT data in the combined_cards list did not have the "nft_price" attribute, I checked if each card had the key "nft_price" and if so, the card was added to the new filtered_combined_cards list.

The filtered_combined_cards list ended up looking like the code snippet below.

[
 {'id': 4, 'nft_price': '0.8', 'rs': 10},
 {'id': 42, 'nft_price': '1.1', 'rs': 5},
 {'id': 174, 'nft_price': '1.4', 'rs': 5},
 {'id': 184, 'nft_price': '1.6' 'rs': 19},
 {'id': 256, 'nft_price': '2', 'rs': 15},
 {'id': 335, 'nft_price': '4', 'rs': 2},
 {'id': 562, 'nft_price': '1.2', 'rs': 2},
 {'id': 584, 'nft_price': '5', 'rs': 14},
 {'id': 597, 'nft_price': '5', 'rs': 17},
 # more NFT data here
]

Once all this data manipulation and list combining was done, the function returned the final list of cards (filtered_combined_cards) that had both rarity score information and an "nft_price" attribute included.

return filtered_combined_cards

For reference, here's the __main__ function in the Python script, which called the get_cards_rarity_score().

for_sale_scraper.py

if __name__ == '__main__':
   scraper = ForSaleNFTScraper();
   cards = scraper.get_cards(max_card_count=200)
   card_data = [] 
   for card in cards:
    info = (scraper.get_nft_data(card))
    card_data.append(info)

   # filter out any extra cards that aren't for sale
   cards_for_sale = scraper.filter_priced_cards(card_data)

   # filter out any extra cards that aren't for sale
   cards_for_sale = scraper.filter_priced_cards(card_data)

Add new object properties to each object in a list

All right, here's the last Python list manipulation tip I'll be sharing in this post: how to add new properties to each object in a list.

After filtering the NFTs to just the ones for sale, and adding the rarity scores from the id_rs_list JSON file, I needed to fetch the current price of 1 BNB compared to US dollars, calculate the current price of each NFT in USD, and calculate the cost per rarity point for each NFT.

Fortunately the cryptocurrency data aggregation site CoinGecko, has a REST API that I could use to get the current market price of BNB cryptocurrency in US dollars, and then calculate the rest of the required data based on the info in my NFT card list.

Here is the add_pricing_to_cards() function I came up with to calculate the prices.

def add_pricing_to_cards(self, card_list):
    """Get current price of BNB and compute cost per rarity point"""

    URL="https://api.coingecko.com/api/v3/simple/price?ids=binancecoin&vs_currencies=USD"
    response = requests.get(URL).json()
    bnb = response['binancecoin']['usd']

    # add the current value of bnb to the card_list 
    cards_bnb_price = [dict(card, bnb=bnb) for card in card_list]

    # compute the current price of usd for each card based on its bnb price       
    cards_with_usd_price= [dict(card, price_usd=round(float(card['nft_price'])*card['bnb'], 2)) for card in cards_bnb_price]

    # compute the current cost usd of each rarity score point         
    cards_with_rs_prices = [dict(card, cost_per_rs=round(card['price_usd']/card['rs'], 2)) for card in cards_with_usd_price]

    return cards_with_rs_prices

In the function, the first thing I did was call the CoinGecko price API to get the current price of BNB in USD.

    URL="https://api.coingecko.com/api/v3/simple/price?ids=binancecoin&vs_currencies=USD"
    response = requests.get(URL).json()
    bnb = response['binancecoin']['usd']

Next, I added the bnb to each object in the input card_list and created a new list named cards_bnb_price.

    # add the current value of bnb to the card_list 
    cards_bnb_price = [dict(card, bnb=bnb) for card in card_list]

After including the current BNB price in USD, I was able to compute the total price in USD for each NFT in the list by multiplying the card's original price in BNB by the current price of BNB in USD.

    # compute the current price of usd for each card based on its bnb price       
    cards_with_usd_price= [dict(card, price_usd=round(float(card['nft_price'])*card['bnb'], 2)) for card in cards_bnb_price]

And I also calculated the price in USD per rarity score point as well, simply by dividing the card's total price in USD by the rarity score number (rs).

    # compute the current cost usd of each rarity score point         
    cards_with_rs_prices = [dict(card, cost_per_rs=round(card['price_usd']/card['rs'], 2)) for card in cards_with_usd_price]

The function then returned the list of cards with the added pricing info, including BNB price, USD price, and USD cost per rarity score point. The final list data looked like this.

[
    {'bnb': 352.44, 'cost_per_rs': 28.2, 'id': 4, 'nft_price': '0.8', 'price_usd': 281.95, 'rs': 10},
    {'bnb': 352.44, 'cost_per_rs': 77.54, 'id': 42, 'nft_price': '1.1', 'price_usd': 387.68, 'rs': 5},
    {'bnb': 352.44, 'cost_per_rs': 98.68, 'id': 174, 'nft_price': '1.4', 'price_usd': 493.42, 'rs': 5}, 
    {'bnb': 352.44, 'cost_per_rs': 29.68, 'id': 184, 'nft_price': '1.6', 'price_usd': 563.9, 'rs': 19},
    {'bnb': 352.44, 'cost_per_rs': 46.99, 'id': 256, 'nft_price': '2', 'price_usd': 704.88, 'rs': 15},
    # more NFT data
]

The add_pricing_to_cards() function is called from the main Python function like so:

for_sale_scraper.py

if __name__ == '__main__':
   scraper = ForSaleNFTScraper();
   cards = scraper.get_cards(max_card_count=200)
   card_data = [] 
   for card in cards:
    info = (scraper.get_nft_data(card))
    card_data.append(info)

   # filter out any extra cards that aren't for sale
   cards_for_sale = scraper.filter_priced_cards(card_data)

   # filter out any extra cards that aren't for sale
   cards_for_sale = scraper.filter_priced_cards(card_data)

   # add rarity scores to all cards in the list by merging them with the id_rs_list
   cards_with_rs = scraper.get_cards_rarity_score(cards_for_sale)

And now that I had all the data that my friend requested for each NFT in the collection for sale on NFTrade, all that was left to do was turn the whole list into a downloadable CSV that would be easy to sort and manipulate. I'll save that for a future post.

Conclusion

When I had to use Python to build a website scraper to get NFT data off of the NFTrade site, I learned a lot of useful new coding tricks along the way.

After I'd managed to scrape the data with the help of Selenium Python, and extract the initial data I needed from each NFT by using WebDriver's XPath, my job was far from complete.

I needed to take the little data I had and combine those NFTs with "rarity scores" in a JSON file, fetch the current market price for BNB cryptocurrency in US dollars, and then compute the total cost of each NFT and cost per rarity point, and as I completed these tasks I learned a heck of a lot about how to work with lists of complex objects in various new ways. And I feel confident these new techniques will help me out in any future Python endeavors I might undertake.

Check back in a few weeks — I’ll be writing more blogs about the problems I had to solve while building this Python website scraper in addition to other topics on JavaScript or something else related to web development.

Thanks for reading. I hope learning to filter, merge, and alter objects within lists in Python proves helpful for you in your own projects.

References & Further Resources

NFTrade website
Python documentation
CoinGecko cryptocurrency tracker and analytics site
First blog post about scraping data from a lazy-loading website using Selenium Python
Follow up blog post about limiting data searches to a particular element on a page instead of the whole page when using XPath

Use Selenium with Python to Target the XPath of a Particular Object

Mon, 18 Dec 2023 00:00:00 GMT

</figure>

Introduction

Earlier this year, I needed to write a program for my friend to collect the data of one of the NFT collections on the NFTrade site. He wanted the following data included: all the NFTs currently for sale in the collection and the price of each NFT in US dollars based on the current market price of the BNB cryptocurrency that the NFT is for sale in, and he wanted it listed out line by line in a CSV file that he could sort and manipulate.

Unfortunately, the NFTrade website does not have a public API so instead of writing a Node.js script to fetch the data via HTTP calls, I built a small script to navigate to the website page and actually "scrape" the data off of it.

Having not written a web scraper before, I chose to write the program in Python as it seems to be a very popular programming language choice for a task such as this. While I built this scraper, the project requirements evolved and got more complex, and I learned a bunch of useful new techniques when using Python, which I'll be sharing in a series of posts over the coming months.

After I'd settled on using the Selenium Python package to start a Selenium WebDriver instance and scrape the data from NFTrade, I hit a snag: I was collecting all the individual NFT data to loop through and pull the details from, but each time the loop ran it only collected the data from the first NFT in the list.

I was stumped and turned to the Internet for help and (once again) Stack Overflow came through for me.

When extracting scraped web data using Selenium WebDriver's XPath to target the data, in order to search inside a particular element instead of the whole document, a period (.) must be added in front of the XPath. I'll show you how to do it in this post.

NOTE: I am not normally a Python developer so my code examples may not be the most efficient or elegant Python code ever written, but they get the job done.

Selenium Python package

For my project, I ended up using the Selenium Python package because it can scrape websites with dynamically loaded data, which is how NFTrade works. A user lands on a collection of NFTs and as they scroll down the page, periodically more NFTs are loaded into the browser window via JavaScript.

Selenium Python operates the Selenium WebDriver software and drives a browser just like a user would, and although its original use was for automated end-to-end testing of software, it can also be used to scrape data off of live web pages.

If you'd like to read more about how to use Selenium Python for this, I encourage you to visit my first blog post on this subject where I go in depth on it.

What is XPath?

Among the many useful methods included with the Selenium Python package are the methods find_element(), find_elements(), and By.XPATH.

The find_element methods do what their name implies: find an element (or elements) given a By strategy and locator.

By accepts element IDs, names, attributes, XPaths, class names, etc. And XPath is a syntax used to navigate through elements and attributes in a standard XML document (webpage).

Due to the NFTrade site's structure, I used XPath expressions to identify all the individual NFTs on the page and scrape the data from each NFT to include later in my CSV file.

The Problem: XPath was targeting the whole document

After I'd written the code to initially fire up my Selenium WebDriver instance, navigate to the NFTrade site, and load the NFTs into the browser that I wanted to scrape the data from, I had a list of NFT info I needed to slim down to just the data points I wanted to include in the CSV.

NOTE: If you'd like to see how to scrape the browser data in-depth, read my first blog post here.

For me, this was things like: the NFT ID number (part of the NFT card's name) and the the NFT's sale price in BNB.

Inside of the __main__ method in my Python script, I'd scraped the data from the webpage with the get_cards() method, and then I wanted to loop through the NFT data I'd collected and extract the data points from each NFT card with my get_nft_data() method.

Here's the __main__ method code for reference:

for_sale_scraper.py

if __name__ == '__main__':
   scraper = ForSaleNFTScraper();
   cards = scraper.get_cards(max_card_count=200)
   card_data = []
   for card in cards:
    info = (scraper.get_nft_data(card)) 
    card_data.append(info)

    # pprint the card data to ensure we're getting the correct data out of each card
    pprint(card_data)

And here is the code for my get_nft_data() method.

def get_nft_data(self, nft_data):
        """Extracts and prints out NFT specific data."""
        nft_name_element = self.driver.find_element(By.XPATH, '//div[contains(@class, "Item_itemName__ckoHR")]')
        nft_name = nft_name_element.get_attribute("innerHTML")
        nft_id = nft_name.partition('#')[-1]

        nft_price_element = nft_data.find_element(By.XPATH, '//div[contains(@class, "Item_itemPriceValueTxt__lblqJ")]')
        nft_price = nft_price_element.get_attribute("innerHTML")
        return {
            'id': int(nft_id), 'price': nft_price
            }

What my get_nft_data() method is supposed to do is:

Use the XPath within each card to get the NFT's name via get_attribute("innerHTML") (innerHTML targets the text content of the element which includes its ID number at the end of the name), partition() the string into a tuple based on the # included in the name, and take the last element in the tuple (which is the ID number).
It should also get the NFT's price (also using get_attribute("innerHTML")) via a second XPath within the NFT card.
Finally return those two elements together as a new object with the keys of id and price.

In theory, I wanted that to happen for every NFT I'd collected into my card_data list. In practice, I got 200 elements that all contained the ID and price data from the very first card in my card_data list.

Not quite what I was hoping for.

The Solution: How to restrict XPath inside a particular element

After several failed variations of the code above and searching through many Stack Overflow posts without success, I finally wrote my own SO post explaining my situation and asking for help from the greater web development community.

If you'd like to see my original Stack Overflow post and the helpful responses provided, here is a link.

Just over 30 minutes after posting my question, a kind soul answered it and got me moving forward again. That's the power of the web dev community at its finest.

Below is the corrected get_nft_data() code that actually gets the data from each individual NFT as the data is looped over. I also added some comments between lines of code to explain what's happening at each step.

def get_nft_data(self, nft_data):
      """Extracts and prints out card specific data."""
      # get full card name "NFT_CARD #1234" by XPATH
      nft_name_element = nft_data.find_element(By.XPATH, './/div[contains(@class, "Item_itemName__ckoHR")]')
      nft_name = nft_name_element.text
      # parse out just ID number from name
      nft_id = nft_name.partition('#')[-1]

      # get nft recently sold value by XPATH
      nft_bnb_sale_price = nft_data.find_elements(By.XPATH, './/div[contains(@class, "Item_itemPriceValueTxt__lblqJ")]')
      
      # if there is a for sale price, take it 
      if nft_bnb_sale_price:
          nft_price = nft_bnb_sale_price[0].text
      else: 
      # if there's no value, just put None in place of a value    
          nft_price = None   

      return {
          'id': int(nft_id), 
          'nft_price': nft_price
          }

In this version of the Python code, there's three main differences.

The first is that instead of using self.driver.find_element, this code is using nft_data.find_element. By substituting nft_data instead of self.driver, it allows the XPath search to be restricted to a particular element.
Second, inside of each find_element method referencing By.XPATH, the XPath being passed has a . in front of it. So '//div[contains(@class, "Item_itemName_ckoHR")]' becomes './/div[contains(@class, "Item_itemName__ckoHR")]'.

The dot (.) further restricts the XPath's search inside of a particular element (or "context node"). If the . is not included, XPath will search the whole document, which is why it was always finding the values from the first NFT element each time the loop ran.
Last, the user suggested I could use .text to get the NFT name and BNB price instead of having to write out the lengthier .get_attribute("innerHTML") each time to reach the text in the NFT, which was a nice improvement in code readability.

The solution also mentioned there was a chance that some of the NFTs collected from the page may not have a price listed (NFTrade displays all NFTs in a collection, not just the ones for sale), and recommended wrapping the code that gets the nft_price in an if / else block so if the price is present, it will be collected and returned, and if it's not, it'll return None in place of the value and not throw an error in the code.

Hence this code for checking sale price:

 # if there is a for sale price, take it 
      if nft_bnb_sale_price:
          nft_price = nft_bnb_sale_price[0].text
      else: 
      # if there's no value, just put None in place of a value    
          nft_price = None

Test the refactored code

With my newly refactored get_nft_data() method at the ready, it was time to test it out in my Python script.

As a reminder, my __main__ method looked like this:

if __name__ == '__main__':
   scraper = ForSaleNFTScraper();
   cards = scraper.get_cards(max_card_count=200)
   card_data = [] 
   for card in cards:
    info = (scraper.get_nft_data(card))
    card_data.append(info)

    # pprint the card data to ensure we're getting the correct data out of each card
    pprint(card_data)

This time, when I ran the script from the command line with python for_sale_scraper.py, here's a screenshot of the output I received.

<figcaption>If you look closely you'll see that each object in this area has a different ID and price in it.</figcaption>

</figure>

As you can see from the image, I got an array of items and each item in the array had a different id and nft_price from the others. Now the get_nft_data() method was working correctly, targeting the next NFT in the card_data array with each successive iteration in the loop and pulling out the data specific to that card.

Success!

With that hurdle overcome, I was ready to move on to the next steps of this project: converting the NFT's BNB prices into the current USD prices and assembling them into a CSV spreadsheet. Those tasks will be covered in detail in upcoming blog posts.

Conclusion

When I was asked to put together a spreadsheet of all the NFTs for sale in a particular collection on NFTrade, I ended up using Python to build a website scraper to accomplish it and learned a lot of new problem-solving techniques in the process.

I managed to load and collect all the NFT data from a web page with the assistance of the Selenium Python package, but I got throughly stuck when trying to iterate through my data to extract the ID and price for each NFT from it.

Luckily, the Stack Overflow community came through for me and taught me about the finer points of using Selenium WebDriver's XPath to target specific elements within a page instead of the whole page when looking for particular pieces of data, which got me unstuck and on my way to building my spreadsheet of data.

Thank goodness for the knowledge sharing of the online web development community - I am truly grateful to be able to turn to them when I've exhausted my own ideas to solve a problem.

Thanks for reading. I hope learning how to restrict XPath searches to a particular element on a page instead of the entire page proves helpful for you in the future like was for me.

References & Further Resources

Selenium Python docs
Selenium WebDriver docs
NFTrade website
XPath documentation
Original Stack Overflow post
Previous blog post about scraping data from a lazy-loading website using Selenium Python

Scrape Data from a Lazy Loading Website with Selenium Python

Sun, 22 Oct 2023 00:00:00 GMT

</figure>

Introduction

A few months ago, my friend wanted me to write a program to collect the data of one of the NFT collections on the NFTrade site, compute the current price of each NFT in US dollars based on the current market price of the BNB cryptocurrency it was listed for sale in, and compile all of the NFTs for sale into a CSV file that he could sort and manipulate.

Unfortunately, the NFTrade website does not have a public API so writing a Node.js script to fetch the data from the API and format it as required was not an option. Instead, I needed to make a site scraper to actually go to the website page and "scrape" the data off of it.

Having not written a web scraper before (and also wanting to make the script easier for my friend to update and run on his own machine), I decided to write the program in Python (it seems to be a very popular programming language choice for a task such as this). Along the way, my little web scraper's requirements evolved and got more complex, and I learned a bunch of useful new techniques about using Python for my project, which I intend to share in a series of posts over the coming months.

My first attempt to scrape the data from NFTrade was unsuccessful beyond locating the first 75 NFTs on the page. I figured out this was because NFTrade (as many other websites do) lazy loads NFTs onto the page 75 at a time: once the user's scrolled down far enough to reach the end of the currently visible items, the site loads the next batch of elements onto the page (essentially a fancier version of pagination). So I needed a way to have my web scraper program collect whatever data was available on the page then scroll down far enough to trigger more data to load and collect that, and rinse and repeat.

After some trial and error, I finally found a working solution with the help of a Python package named Selenium Python, and I'll share with you today how to write your own Python script to scrape data from a lazy loading website with Selenium WebDriver.

NOTE: I am not normally a Python developer so my code examples may not be the most efficient or elegant Python code ever written, but they get the job done.

Selenium with Python package

There are a few different popular Python packages available for web scraping which I tried before reaching for Selenium, but I had an issue with them in that they only worked for static websites that were generated at build time, not for sites that are generated on the client-side via JavaScript, like NFTrade is.

To that end, I had to do a little digging to find a package that could work with scraping sites with dynamically loaded data, and I ran across the Selenium Python package during my investigation.

Selenium Python is a Python-based API that allows users to write scripts or automated tests using Selenium WebDriver in an intuitive, Python-flavored way. And Selenium WebDriver is a software that can drive a browser natively, as a user would, either locally or on a remote machine. Originally created back in 2004, some version of Selenium has been around for years and is considered one of the earliest versions of automated testing that emulates user actions on a web page (commonly known today as end-to-end testing).

The cool thing about WebDriver though, is that its uses span beyond automation testing, as scripts can actually be written to scrape data off of live web pages, and that's just what I ended up doing in my Python script, so let's get started.

Install Selenium Python in the Python project

As with most projects, the first thing to do is add the Selenium Python package to the Python project. The easiest way is to use pip to install the Selenium package.

Assuming you have pip on your machine, at the root of your Python project folder, run the following command from a terminal.

pip install selenium

Then, add the selenium package to your requirements.txt file so anyone downloading the repo in the future can install all the necessary project dependencies.

requirements.txt

selenium

And that's all it takes to be ready to use WebDriver in your Python script. Simple enough.

Import Selenium WebDriver into Python script

After adding the Selenium Python bindings to the project, it's time to import Selenium's WebDriver and some of its helpful configuration options to the actual Python script that does the website scraping. I named my file for_sale_scraper.py since I was specifically looking for NFTs that are for sale (not all of the NFTs listed on NFTrade are - some are just visible but not actually available to purchase), but you can choose any sort of file name that makes sense for you.

Below are the imports I added to my file. I'll break down what each one is doing below.

for_sale_scraper.py

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By

The very first import line brings in the selenium.webdriver module and provides all the WebDriver implementations.

from selenium import webdriver

Next, as I chose to use Chrome as the browser I wanted WebDriver to interact with (Selenium supports Firefox, Chrome, Edge, and Safari browsers), I imported the Options class from the selenium.webdriver.chrome.options module. This allowed me to add specific config details about how I want the Chrome browser to be set up when the Python script runs against it: things like headless mode or disable extensions, etc.

from selenium.webdriver.chrome.options import Options

I'll cover the arguments I passed here in detail in the next section.

WebDriverWait, added in the third line of imports, is part of the special sauce that makes WebDriver a good solution for sites like NFTrade that dynamically fetch data on the client side: it allows for implicit and explicit wait times before trying to locate an element on the page, which gives the browser time for data to come back from the server and populate in the DOM.

from selenium.webdriver.support.wait import WebDriverWait

This type of wait is an "explicit wait", meaning I manually set a period during which the code will wait before continuing to try and execute.

And finally, there is the import for By. By is what allows me to locate elements on the page - it is immeasurably useful and powerful.

The By class accepts element IDs, names, attributes, XPaths, link text, tag names, class names, and CSS selectors just to name a few, and once again, it is a key player when it comes to scraping data off of the web page, as I'll demonstrate soon.

from selenium.webdriver.common.by import By

Right, all the Selenium WebDriver imports are now present in the Python file, time to initialize them and get to work.

Add methods to scrape data and lazy load more data

Before WebDriver can begin scraping the data from NFTrade, an instance of the browser that WebDriver will interact with must be instantiated and the proper options supplied to it.

1. Initialize the Selenium WebDriver instance

In my attempt to try to follow good Python coding practices (again, disclaimer: I don't write Python as my primary coding language), I created a class for the the file named class ForSaleNFTScraper, and created an __init__() method immediately inside of it where I created the Chrome WebDriver instance that the whole script will be able reference in the remainder of its methods.

class ForSaleNFTScraper:
    def __init__(self):
        options = Options()
        options.add_argument('--headless')
        options.add_argument('--start-maximized')
        self.driver = webdriver.Chrome(options=options)
        self.wait = WebDriverWait(self.driver, 5)

# more code here

The first thing I did inside of the __init__() method was to add a couple of Chrome browser configs via the Options import from the last section by declaring a new options variable.

    options = Options():

Since I wanted this script to run without actually opening a browser window on the user's local machine, I added the config argument of --headless and the argument of --start-maximized, so the (unseen) window would take up as much screen size as was available (and hopefully load as many NFTs as quickly as possible by doing so).

    options.add_argument('--headless')
    options.add_argument('--start-maximized')

Then I passed the new options object to the instance of webdriver.Chrome, which was set to the variable of self.driver (self is a variable accessible throughout the rest of the methods within this ForSaleNFTScraper class), and instructed the new WebDriver to wait for 5 seconds after startup (which would presumably give it time to go to the specified NFTrade web URL and load the data onto the page before attempting to scrape it).

    self.driver = webdriver.Chrome(options=options)
    self.wait = WebDriverWait(self.driver, 5)

There's plenty happening in that first method, but it's all pretty straightforward once you go through the code line by line and understand what the arguments mean to the Chrome WebDriver instance, and why it's doing what it's doing. Now that the WebDriver instance was configured and ready to go, I could write the code fetching the NFT card data, and lazy loading more data once the end of the currently visible info was reached.

2. Write the get_cards() and get_current_card_count() methods

This is where the code really starts to get interesting in my opinion, because it's where I learned to collect whatever data was currently visible in a (headless) browser and then load more data to add to the list. Pay close attention, because this is where the lazy loading code resides that gets more and more data onto the page.

def get_current_card_count(self):
    """Get the count of cards loaded into list of cards."""
    return len(self.driver.find_elements(By.XPATH, '//div[contains(@class, "Item_itemContent__1XIcH")]'))    

def get_cards(self, max_card_count = 500):
    """Extract and returns card ID and price."""
    URL="https://nftrade.com/collection/[NFT_COLLECTION_NAME]"
    self.driver.get(URL)
    last_card_count = 0
    # loops through lazy loading cards on site until max_card_count number reached 
    while last_card_count < max_card_count:
        self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        time.sleep(3)
        last_card_count = self.get_current_card_count()
    # grab all the cards now loaded into the browser by XPATH
    cards = self.driver.find_elements(By.XPATH, '//div[contains(@class, "Item_itemContent__1XIcH")]')
    return cards

# more code here

Ok, here we go.

For starters, there are two methods that I'm displaying in the code snippet here. The first method, get_current_card_count() is how I keep track of how many NFT cards in a collection are currently visible on the screen.

As I've said, NFTrade lazy loads its NFT collections onto a site to make initial page load quicker, and when a user scrolls down to the end of the currently loaded batch of elements, the NFTrade page then triggers to load more cards into the DOM at that point in time.

The second method is get_cards(), which handles going to the NFTrade collection URL and scraping all the available card data. It relies on get_current_card_count() to help it know to load more NFT cards until the desired number of cards has been loaded in the browser to scrape data from.

get_cards() method

I'll talk about get_cards() first as it's the more complicated of the two methods. The first thing the method does is declare a new variable named URL - this variable is set to the URL of the NFTrade collection page I want WebDriver to navigate to and scrape the data from. I used the Selenium WebDriver driver.get() method to navigate to the page given by the URL.

    URL="https://nftrade.com/collection/[NFT_COLLECTION_NAME]"
    self.driver.get(URL)

After navigating to the proper URL, I created a variable called last_card_count and set it equal to 0: this variable will be used to track how many NFTs are currently visible on the page and compare it to the max_card_count variable passed to the get_cards() method (if a number isn't passed for max_card_count it defaults to 500).

Below is the key code to lazy loading more and more data in the browser

Inside of get_cards(), there's a while loop set up to compare the last_card_count and max_card_count variables. As long as last_card_count is less then max_card_count, the loop will run, and each time it executes WebDriver uses the driver.execute_script() method to scroll down the page, wait for 3 seconds (allowing more cards to load onscreen), and then updating the last_card_count variable equal to the new amount of cards on the page using the get_current_card_count() method.

NOTE: The window.scrollTo() method is critical

driver.execute_script() allows for the synchronous execution of JavaScript in the current window, so when you see the code self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);"), what's happening is that WebDriver is using the JavaScript window.scrollTo() method to scroll the browser all the way to the bottom of the page (that's why document.body.scrollHeight is present - it's a measurement of the height of the whole document.body page element), which triggers the page to load more NFT cards into view.

    last_card_count = 0
    # loops through lazy loading cards on site until max_card_count number reached 
    while last_card_count < max_card_count:
        self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        time.sleep(3)
        last_card_count = self.get_current_card_count()

And this is a perfect time to segue into discussing the get_current_card_count() method, which is short and sweet.

get_current_card_count() method

This method exists simply to find the count of the current elements loaded in the browser, and it does so by combining the WebDriver find_elements() method with the By.XPATH element locator method.

Due to how the NFTrade site is built, there are no easily identifiable classes, IDs, or other consistent ways to identify all the cards on the page, so I had to resort to XPath expressions to identify each element and include it in my count to update the last_card_count variable. I cobbled together the XPath below by using my Chrome DevTools to inspect the elements on the page and construct the XPath from there through trial and error.

NOTE: What is XPath?

If you're unfamiliar like I was, XPath is a syntax that can be used to navigate through elements and attributes in a standard XML document (or webpage). The link I provided to W3Schools has some good examples of what typical XPath expressions look like and how to interpret them.

So the code inside of the get_current_card_count() method is just the one line of code:

return len(self.driver.find_elements(By.XPATH, '//div[contains(@class, "Item_itemContent__1XIcH")]'))

In the code snippet, I'm getting the count (using the build-in Python method len()) of all the elements on the page that match the XPath of a <div> containing the class of "Item_itemContent__1XIcH", because each NFT on the page is wrapped by that <div> with that class. It's not the prettiest thing to read and understand, but it gets the job done.

And finally, jumping back to the get_cards() method again, once the last_card_count variable has been updated and surpassed the max_card_count variable (i.e. enough NFT cards are loaded into the browser), the while loop ends, and all the cards on the screen are targeted (using the very same XPath used in the get_current_card_count() method, I might add) and set equal to the cards variable defined at the top of the get_cards() method. That variable then gets returned to the __main__ method running the whole script, which I'll cover next.

    # grab all the cards now loaded into the browser by XPATH
    cards = self.driver.find_elements(By.XPATH, '//div[contains(@class, "Item_itemContent__1XIcH")]')
    return cards

There's quite a bit going on here, but hopefully it makes more sense now what these methods are doing. Time to test out this lazy loading script functionality and see how WebDriver does.

3. Run the Python script

All right, now that all the code and logic to load multiple sets of NFTrade cards into the browser and collect the data has been written, it's time to run the code.

To do that, I declared a __main__ method at the bottom of the file which can be started from the terminal with the following command.

python for_sale_scraper.py

Here is what __main__ method includes.

if __name__ == '__main__':
   scraper = ForSaleNFTScraper();

  # get all the cards from nftrade site
   cards = scraper.get_cards(max_card_count=200)

   # pprint the card data to ensure we're getting data
   pprint(cards)

   print("Total cards collected:", len(cards))
   # more code here

The first thing the method does is create a new instantiation variable named scraper by calling the ForSaleNFTScraper() class. It then proceeds to fetch all the card data and set it equal to a variable named cards by calling the method scraper.get_cards(max_card_count=200) and supplying a max_card_count variable of 200.

After this step, as a sanity check, I used the Python pprint() and print() methods to print out all the card data and a count of the total cards fetched by the get_cards() method, and ensure all the info I needed to include in the CSV (NFT price, NFT ID, etc.) was available to me. Here's a screenshot of some of the data printed out in my console helping me know my code is doing what I expect.

<figcaption>Here is what the raw NFT card data gathered from the get_cards() method looks like printed in the terminal.</figcaption>

</figure>

<figcaption>Since I set my max_card_count to 200, but NFTrade loads NFTs in batches of 75 at a time, it makes sense that the total count of NFTs scraped off the page equals 225.</figcaption>

</figure>

And after verifying the right data's there (and the right amount of data as well), I continued on extracting the data, calculating the current price in USD for each NFT, and assembling a CSV of all the data. But I'll save those steps for future blog posts.

Conclusion

Building a Python-based website scraper to create a CSV of NFTs available for sale on NFTrade was a unique challenge I learned a lot of new things from.

After my first attempt failed due to NFTrade dynamically lazy loading NFTs in batches of 75 onto the page as a user scrolled further down, I had to come up with a more creative solution that would allow me to trigger the site to load more cards on the page first, then grab the data on the cards for sale.

I found the solution I was looking for with the help of a Python package called Selenium Python. Selenium Python is a powerful Python-based API that allows users to write scripts or automated tests leveraging Selenium WebDriver. And it was up to the task at hand: with just a few methods I was able to specify as many NFTs as I wanted loaded on the page before scraping and collecting all their data all at once.

Thanks for reading. I hope seeing how to make a Python Selenium WebDriver load data onto a dynamic webpage before scraping it comes in handy for you in the future.

References & Further Resources

Copy Files from One Repo to Another Automatically with GitHub Actions

Sun, 13 Aug 2023 00:00:00 GMT

</figure>

Introduction

Welcome to the third (and final) installment in my series about the many things I learned while building my first open source API library for Blues, the Internet of Things startup I work for.

It was a good challenge and a great learning opportunity for me.

I automated as many pieces of the library's maintenance and deployment to as I could with the help of GitHub Actions workflows, one of them being: copy the openapi.yaml file from the Blues cloud repo, Notehub, whenever that file was updated, and push it into a feature branch in the Notehub JS repo (the name of the API library).

Interacting with two different repos in one workflow might sound tricky, and at first it was, but it is possible and not too bad once you see how it can be done.

In this tutorial, I'll demonstrate how to create a GitHub Actions workflow that automatically copies a file from one repo to another and pushes the changes to another repo inside a feature branch.

I know the use case sounds a bit specific, but it came in really handy for me, and you never know when it might for you too.

Notehub JS

Before we get to the actual GitHub Actions workflows, I'll give you a little background on the Notehub JS project.

This section outlines the folder structure for the repo in case you want to explore it in GitHub, if you just want to get to the solutions, feel free to jump down to the next section.

Notehub JS is a JavaScript-based library for interacting with the native Notehub API, and it is actually generated from the Notehub project's own openapi.yaml file, which follows the OpenAPI specification standards.

The OpenAPI Generator CLI is a tool that uses the openapi.yaml file to create an entire API library complete with documentation, models, endpoints, and scripts to package it up for publishing as an npm module. The end library that is published to npm is a subfolder inside of the main Notehub JS repo. At the root of the project are the openapi.yaml file, the GitHub Actions workflows, and a few other config files.

Here's a simplified view of the Notehub JS repo's folder structure:

root/
├── .github/ 
| ├── workflows/
| | ├── create-pr.yml 
| ├── PULL_REQUEST_TEMPLATE.md
├── src/ <- this is the folder generated by the OpenAPI Generator CLI
| ├── dist/
| ├── docs/
| ├── src/ 
| | ├── api/ 
| | ├── model/
| | ├── index.js 
| openapi.yaml
| config.json
| package.json

Based on the diagram above, the openapi.yaml file that the library is generated from lives at the root of the repo, and the src/ folder is what holds all the Notehub API JavaScript code that powers the Notehub JS library.

This openapi.yaml file gets copied from the Notehub repo when changes are made to it. And changes are made to that file whenever the Notehub API is updated with new features and functionality, so making sure that the Notehub JS library keeps up with the new additions to the original API it's based on is important.

Now that I've explained more about the Notehub JS repo and why keeping it in sync with the Notehub API is important (and depends on the openapi.yaml file), we can get down to the business of ensuring the file gets copied to this repo whenever it's updated.

Looking for more details about Notehub JS?

Check out my first blog post about how to use GitHub Actions to automatically publish new releases to npm - I do a fairly deep dive on Notehub JS there.

Create a GitHub Actions workflow to run when the `openapi.yaml` file is updated

First things first: creating a new GitHub Actions workflow that's triggered when the openapi.yaml file changes in the Notehub repo.

Need a refresher on GitHub Actions?

If you want a quick primer on what GitHub Actions are, I recommend you check out a previous article I wrote about them here.

Below is the final version of the copy-openapi-file.yml file, which lives inside of the ./github/workflows/ folder. I'll walk through each step of the script below.

.github/workflows/copy-openapi-file.yml

name: Copy updated OpenAPI file

# only run this workflow when the openapi file has changed on the master branch
on: 
  push:
    branches: 
      - master
    paths:
      - 'notehub/api/openapi.yaml'

jobs:
  copy_openapi_file_to_notehub_js: 
    runs-on: ubuntu-latest
    steps:
    # check out notehub project
    - name: Check out Notehub project
      uses: actions/checkout@v3
    # check out notehub-js project using token generated in that project to successfully access it from inside the Notehub GitHub Action workflow
    - name: Check out Notehub JS project
      uses: actions/checkout@v3
      with:
        repository: blues/notehub-js
        path: ./notehub-js
        token: ${{ secrets.NOTEHUB_JS_TOKEN }}
    # make a copy the openapi file from notehub project
    - name: Copy OpenAPI file
      run: bash ./.github/scripts/copy-openapi-file.sh
      env:
        DESTINATION_PATH: ./notehub-js/
    # make a branch in notehub-js repo and push the copy of the openapi file there
    - name: Push to notehub-js repo
      run: bash ./.github/scripts/push-openapi-to-notehub-js.sh
      env:
        BRANCH: feat-openapi-update

Each GitHub Actions workflow needs a name, and I try to choose descriptive ones like this one: Copy updated OpenAPI file.

Then, I want this workflow to only run when the openapi.yaml file in the master branch of the project is changed. master is the branch that gets deployed to production, and in my case it's a safe assumption that when there's a change to this file in master, those changes are going to prod, which is when I want to copy the file over to the Notehub JS repo so it can reflect those same changes.

The code snippet below does just that.

on: 
  push:
    branches: 
      - master
    paths:
      - 'notehub/api/openapi.yaml'

on is how a workflow is triggered.
push is the event that triggers the workflow.
branches defines which branch names this workflow should run for.
paths is another filter to even more finely determine when the workflow runs.

What it boils down to is: when there's a change to the master branch in the repo for the openapi.yaml file at this particular file path, run the GitHub Actions workflow.

From there, the copy_openapi_file_to_notehub_js job runs. There's only one job in the jobs section of this script, but if there were multiple jobs, they'll run sequentially.

The create_pr_repo_sync job defines that it runs on the latest version of Ubuntu in runs-on.

And finally, I get to the steps section. Here's how it goes:

Check out the Notehub code so the workflow can access it using the GitHub Action actions/checkout@v3.
Check out the Notehub JS code so the workflow can access it as well using the GitHub Action actions/checkout@v3.
- NOTE: Notice that in this second checkout action I had to include a with parameter that allows me to specify another repo to check out by providing a repository, path, and token. The token is a GitHub Personal Access Token (PAT) required when checking out other libraries beyond the current one the workflow is operating within.
Run a shell script named copy-openapi-file.sh with the GitHub Actions environment variable DESTINATION_PATH passed to the script.
Run another shell script named push-openapi-to-notehub-js.sh with the GitHub Actions environment variable BRANCH passed to the script.

In the following sections I'll go through what those two shell scripts do in detail.

Ok, so the workflow file is done. Time to get into the details of the two scripts doing the heavy lifting.

Write one shell script to copy the `openapi.yaml` file to the Notehub JS repo

Premade GitHub Actions can do many things, but something as specific as copying a file from one repo to another is a bit beyond them.

Luckily, GitHub Action workflows can run shell scripts, so if you can use a scripting language like PowerShell or Bash, you're in luck.

Here is the Bash script I wrote to copy the file from the Notehub repo to the Notehub JS repo.

copy_openapi_file.sh

#!/usr/bin/env bash
# bash boilerplate
set -euo pipefail # strict mode
readonly SCRIPT_NAME="$(basename "$0")"
readonly SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
function l { # Log a message to the terminal.
    echo
    echo -e "[$SCRIPT_NAME] ${1:-}"
}

# File to copy from Notehub
OPENAPI_FILE=./notehub/api/openapi.yaml

# if the file exists in Notehub, copy it to Notehub-JS repo
if [ -f "$OPENAPI_FILE" ]; then
    echo "Copying $OPENAPI_FILE"
    cp -R ./notehub/api/openapi.yaml $DESTINATION_PATH
fi

echo "OpenAPI file copied to $DESTINATION_PATH"

The 9 lines at the top of the file are Bash script set up, so don't worry too much about what's happening there.

The first line to pay attention to is where the OPENAPI_FILE variable is defined with a path to the openapi.yaml file in the Notehub repo.

After identifying the file to copy from the Notehub repo, a standard Bash if statement checks if the file exists in the Notehub repo (just to be safe), and if it does, it copies the file recursively using cp -R to the $DESTINATION_PATH variable that was provided by the GitHub Actions environment variable.

In this case, the $DESTINATION_PATH is pointing to the root folder of the Notehub JS project where the file should be copied to, but you could specify it copy the file anywhere within the receiving repo that makes sense.

During the copying process and when the script finishes copying the file I added a few echo commands to print out useful messages in the logs to let me know things are going to plan.

And that's the first shell script that actually copies the file that will be taken from one repo to the other. I hope it's not too complicated once the actions in the script are broken down.

Now, it's time to add that file to the Notehub JS repo.

Use a second shell script to push the copied file to a feature branch in the other repo

When I checked the GitHub Marketplace for a ready-made action to create a new feature branch in the repo receiving the copied file, there were a few to choose from, but I wanted more control over what was happening than they offered.

I wanted an action that would:

Create a new branch in the Notehub JS receiving repo if it didn't exist, add the copied file, and then push that branch to GitHub.
Check out the already existing branch in the Notehub JS repo, overwrite any previous file changes in the branch, and then push the new file changes in that branch back to GitHub.

Here's the bash script that I came up with to handle those two scenarios.

push-openapi-to-notehub-js.sh

#!/usr/bin/env bash
# bash boilerplate
readonly SCRIPT_NAME="$(basename "$0")"
readonly SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
function l { # Log a message to the terminal.
    echo
    echo -e "[$SCRIPT_NAME] ${1:-}"
}

# move to the root the notehub-js repo
cd "./notehub-js"
echo "Open root of Notehub JS repo"

# check if there's already a currently existing feature branch in notehub-js for this branch
# i.e. the altered openapi.yaml file's already been copied there at least once before
echo "Check if feature branch $BRANCH already exists in Notehub JS"
git ls-remote --exit-code --heads origin $BRANCH >/dev/null 2>&1
EXIT_CODE=$?
echo "EXIT CODE $EXIT_CODE"

if [[ $EXIT_CODE == "0" ]]; then
  echo "Git branch '$BRANCH' exists in the remote repository"
  # fetch branches from notehub-js
  git fetch
  # stash currently copied openapi.yaml
  git stash
  # check out existing branch from notehub-js
  git checkout $BRANCH 
  # overwrite any previous openapi.yaml changes with current ones
  git checkout stash -- .
else
  echo "Git branch '$BRANCH' does not exist in the remote repository"
  # create a new branch in notehub-js 
  git checkout -b $BRANCH
fi

git add -A .
git config user.name github-actions
git config user.email github-actions@github.com
git commit -am "feat: Update OpenAPI file replicated from Notehub"
git push --set-upstream origin $BRANCH

echo "Updated OpenAPI file successfully pushed to notehub-js repo"

Once more, the first 9 lines of code are boilerplate for Bash; no comments necessary.

The first thing this script does is situate itself in the root of the Notehub JS folder just as you would when navigating around a computer via the command line: cd "./notehub-js".

Once there, it checks if the feature branch environment variable passed in by the GitHub Action $BRANCH already exists in the Notehub JS repo with this one-liner.

git ls-remote --exit-code --heads origin $BRANCH >/dev/null 2>&1

Essentially, this code is:

Getting a list of references in the remote repository with git ls-remote,
Checking if env var $BRANCH (passed from the GitHub Action) exists only in the refs/heads of the origin remote with --heads
- (i.e. does the branch name already exist in the GitHub repo where it keeps track of the names of all of its branches),
And returning an --exit-code with status "2" when no matching refs are found in the remote repo and a status of "0" if a matching ref is found.

When the exit code is returned, it's set equal to the variable $EXIT_CODE.

Then, an if/else statement checks what $EXIT_CODE equals.

EXIT_CODE = "0"

If the exit code is 0, that means the branch already exists in the repository; this could happen if multiple changes are made to the openapi.yaml file before the feature branch in Notehub JS gets merged in and deleted.

When that happens the following steps take place:

The branches for the Notehub JS repo are fetched via git fetch,
The current changes (the freshly copied openapi.yaml file) are stashed with git stash,
The existing branch of the same name as the $BRANCH env var is checked out locally,
The stashed changes are applied to that branch overwriting whatever was there before with git checkout stash -- ..

EXIT_CODE = "2"

If $EXIT_CODE is equal to 2, that means the branch doesn't already exist remotely in GitHub and the script can simply checkout a new branch with the name in $BRANCH via git checkout -b $BRANCH.

Finally, after all this is completed, a standard set of git commands follows:

Use git-add -A . to stage all changes in the working directory,
Configure a username and email address for who's doing the commit with git config user.name and git config user.email,
Add a commit message with git commit -am "Standard commit message here",
Lastly, push it to GitHub with git push --set-upstream origin $BRANCH.

At this point, you should be good to go. The file should be successfully copied over to the other repository.

NOTE: Inspiration for this second Bash script was initially based on this blog post by Remarkable Mark.

Check that the workflow runs

The simplest way to test this new functionality is just to keep an eye out for changes in the main branch of the repo that would trigger the workflow to run and see if it works.

When you name the workflow a descriptive name, it's easy to find it amongst the various GitHub Action workflows the repo may have.

Here's a screenshot of the workflow runs of the Copy updated OpenAPI file job inside the GitHub repo's Actions page.

And if you want to dig in further to any of the runs to check the logs and job steps (especially handy for debugging purposes), just click on a workflow run, then the name of the job, and inside the job you can expand out each step.

Here's another screenshot of the Push to notehub-js repo step expanded to see exactly what the Bash script in that step is logging to the console via echo.

And you're done! The workflow's running, the steps are successfully completing, and, most importantly, the file from the Notehub repo is successfully copied to the Notehub JS repo. Mission accomplished.

Conclusion

My first foray into building an open source software library on behalf of one of my company's APIs was a unique challenge that taught me a lot. Especially as I got to leverage GitHub Action workflows to automate as many parts of the process as I could to make ongoing maintenance and upkeep easier.

One such task that I automated involved copying a file from one repo into another whenever a change was made to that particular file. And while I was able to leverage a few pre-existing GitHub Actions, I also had to write a couple custom Bash scripts for some of the trickier, more-unique-to-my-use-case steps.

Fortunately, GitHub Action workflows allows for both types of steps to be combined in a workflow, making for some very powerful, turnkey solutions that did just what I needed.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading. I hope learning to copy a file from one repo to another automatically with GitHub Actions workflows is as helpful for you as it has been for me. Enjoy!

References & Further Resources

Previous OSS article using GH Actions to publish a package to npm
Previous OSS article using GH Actions to create a new PR branch
Notehub JS GitHub repo

Automatically Create a Pull Request Against a Feature Branch with GitHub Actions

Sun, 11 Jun 2023 00:00:00 GMT

</figure>

Introduction

This blog post is the second in a short series I'm writing about the many things I learned in the course of building my first open source API library for the IoT startup I work for, Blues.

It was a great learning experience for me and a new, unique challenge because I needed to do the following things:

Make a copy of the openapi.yaml file from the Blues cloud's repository, Notehub, whenever the file was updated.
Open a new pull request against the Notehub JS repo with the copy of the openapi.yaml file.
Generate a fresh version of the Notehub JS API library based on the that file via the OpenAPI Generator CLI.
And publish the updated Notehub JS library to npm.

And I wanted to automate as many of these steps as possible through the use of GitHub Actions workflows.

Since I didn't know how often the Notehub's openapi.yaml file would be updated, I needed a way to notify myself when a new version of the Notehub's API file needed review in the Notehub JS repository. The best solution I could think of was to open a new pull request in the Notehub JS repo after copying the updated openapi.yaml file into a feature branch and tagging myself to review it.

Today, I'll walk through how to use a GitHub Actions workflow to create (or update) a pull request whenever a new feature branch is made in that repository.

Notehub JS

Before we get to the actual GitHub Actions workflow, let me give you just a little background on the Notehub JS project because it's a bit different than most.

This section helps explain the folder structure for the repo in case you want to explore it in GitHub, if you just want the solutions, feel free to jump down to the next section.

Notehub JS is a JavaScript-based library for interacting with the native Notehub API, and it's generated from the Notehub project's own openapi.yaml file, which follows the OpenAPI specification standards.

The OpenAPI Generator CLI is a tool that can use the openapi.yaml file to create an entire library complete with documentation, models, endpoints, and scripts to package it up for publishing as an npm module. The end library that I care about publishing to npm is a subfolder inside of the main Notehub JS repo. At the root of the project are the openapi.yaml file, the GitHub Actions workflows, and a few other config files.

Here's a simplified view of the Notehub JS repo's folder structure:

root/
├── .github/ 
| ├── workflows/
| | ├── create-pr.yml 
| ├── PULL_REQUEST_TEMPLATE.md
├── src/ <- this is the folder generated by the OpenAPI Generator CLI
| ├── dist/
| ├── docs/
| ├── src/ 
| | ├── api/ 
| | ├── model/
| | ├── index.js 
| openapi.yaml
| config.json
| package.json

As you can see in the diagram above, the openapi.yaml file that this library is generated from lives at the root of the repo, and the src/ folder is what actually holds all the Notehub API JavaScript code that powers the Notehub JS library.

This openapi.yaml file is what gets copied from the Notehub repo when changes are made to it. And changes are made to the file whenever the Notehub API is updated with new features and functionality, so making sure that the Notehub JS library keeps up with the new additions to the API it's based on is important.

Now that I've explained a bit more about the Notehub JS repo and why keeping it in sync with the Notehub API is important, we can get down to the business of automating PRs for this repo.

Looking for more details about Notehub JS?

Check out my previous blog post about how to use GitHub Actions to automatically publish new releases to npm - I do a fairly deep dive on Notehub JS there.

Create a standard PR template for the repo

When I was just a few years into my own web development career, I was introduced to GitHub PR templates, an easier way to keep pull requests uniform for a repository worked on by multiple developers. It made the task of creating decent PRs so much better.

There is not a team or repo I join now without adding a ./github/ directory folder and PULL_REQUEST_TEMPLATE.md file if it doesn't already exist.

For the Notehub JS repo, as it's mostly autogenerated, I included a couple of sections to fill in: Problem Context: a brief description of the updates in the PR, and Changes: what code changes were actually made in the PR. Simple enough for any dev to fill out.

PULL_REQUEST_TEMPLATE.md

# Problem Context

## Changes

That's all I wanted to add to the repo before focusing on the GitHub Actions workflow itself. We'll get to it next.

Set up a GitHub Actions workflow to automatically create pull requests

Need a refresher on GitHub Actions?

If you want a quick primer on what GitHub Actions are, I recommend you check out a previous article I wrote about them here.

For this particular workflow, I was able to string together a few premade GitHub Actions to do just what I needed:

Create a new pull request whenever a new feature branch was pushed to the GitHub repository.
Use the PULL_REQUEST_TEMPLATE.md file to format the PR.
Notify me when a PR is there for review.

Here's what the final create-pr.yml file looks like inside of the ./github/workflows/ directory. I'll dissect it below.

create-pr.yml

name: Automatically create / update pull request

# run this workflow only on new feature branches, not when they're merged to main
on:
  push:
    branches-ignore:
      - "main"

jobs:
  create_pr_repo_sync:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Create pull request
        id: open-pr
        uses: repo-sync/pull-request@v2
        with:
          destination_branch: "main"
          pr_title: "feat: PLACEHOLDER TITLE"
          pr_template: ".github/PULL_REQUEST_TEMPLATE.md"
          pr_reviewer: "paigen11"
          pr_draft: true

Each workflow file needs a name, so I chose to name this one: Automatically create / update pull request.

This workflow should be triggered whenever a new branch is pushed to the repo but it should ignore the "main" branch. "main" is the main branch of this repo that gets published to npm and doesn't need a PR to be created for it.

on:
  push:
    branches-ignore:
      - "main"

on is how a workflow is triggered.
push is the event that triggers the workflow.
branches-ignore is how to exclude the workflow from running on certain branch patterns. This gives us more fine-grained control of when the workflow should run.

Then the jobs section runs inside of the workflow. This particular script only has one job, create_pr_repo_sync, but if there's multiple jobs, they'll run sequentially unless otherwise specified.

The create_pr_repo_sync job defines that it runs on the latest version of Ubuntu in runs-on.

Finally, I get to the steps.

The steps are as follows:

Check out the code so the workflow can access it using the GitHub Action actions/checkout@v3.
Create a pull request using the GitHub Action repo-sync/pull-request@v2
Pass the PR action a custom pr_title, pr_template, and pr_reviewer. Set it to be created as a pr_draft (I like PRs to be drafts until I've looked them over and know they're ready for review), and open the PR branch against the destination_branch: "main".

In the pr_template section, I pass the path to the PR template file I made in the previous section, and for the pr_reviewer I add my own GitHub username so that I get an email notification when the new PR is created in GitHub.

Test out the workflow

If you want to test this functionality out, create a new local branch of the repo, make a change in the branch, and push it to GitHub. When that new branch registers, the GitHub Actions workflow will be triggered to run.

If you visit the Actions page in the GitHub repo, you should see the "Automatically create / update pull request" job running.

And after the action finishes running, there should be an email sent to the folks tagged as the PR reviewers in the GitHub Action (me, in this case).

And with that, the workflow working has been confirmed, and the reviewers know it's time to take action in that repo.

Conclusion

When I built my first open source software library, I learned a lot. My software is a JavaScript library based on my company's API, Notehub, so every time the Notehub API gets updated, the JS library based off of it, needs to be updated too.

I needed a programmatic way to let myself know whenever new changes were made to the JS repo that needed to be reviewed.

The easiest way I could think to accomplish this was by automating the pull request creation so that when a new feature branch was pushed to the repo, it opened a PR and tagged myself as the reviewer.

Luckily, a GitHub Action existed for just such a scenario, allowing for designating a PR template, a PR reviewer, giving the PR a placeholder title, and a whole host of other configurations. The workflow made it quite straightforward, and gave me the peace of mind that whenever changes are made that I need to review, I get an email to look at what's changed.

Thanks for reading. I hope learning to use GitHub Actions workflows to automate a new PR when a feature branch is pushed to a repository proves useful. Enjoy!

References & Further Resources

Previous OSS article using GH Actions to publish a package to npm
Notehub JS GitHub repo
GitHub PR templates
Pull request GitHub Action

Use GitHub Actions to Automatically Publish a Repo Subfolder as an npm Library

Wed, 12 Apr 2023 00:00:00 GMT

</figure>

Introduction

I've said it before, and I'll say it again, working as a software engineer for a startup is fun and challenging because of the sheer number of different things I get to do on a regular basis.

I work for an Internet of Things startup called Blues which specializes in getting IoT data from devices in the real world into the cloud via cellular. To help show our customers all the possibilities of what they can do with our hardware (Notecards) and our cloud (Notehub), a group of my coworkers and I have been building lots of JavaScript-based web apps to monitor and display IoT data for a variety of different use cases.

A little more about Notecards and Notehub.io

Notecards are low-power, 30mm x 35mm, prepaid cellular-connected devices designed to integrate with any IoT device, and Notecards know, out of the box, how to connect with the Notehub cloud to send as well as receive data from it.

Notehub.io is a cloud-based service for securely connecting to Notecard devices and storing data from them before sending it on to your cloud application of choice.

It quickly became apparent that rewriting the same HTTP calls in multiple projects to fetch data from the native Notehub API endpoints wasn’t easy to reuse or keep up to date as the API continues to grow and evolve.

So we created our own open source, JavaScript-based library for the Notehub API and published it on npm for anyone to use for free. Notehub JS is designed to get you connected to the Notehub API quickly, and allow you to access all of the API routes relevant to interacting with Notehub in a JavaScript-friendly way.

Figuring out how to build an open source library like this and publish it to npm was a first for me, and I learned a lot of interesting things along the way that I'd like to share with you all over the course of a few blog posts in the hope that it might help you if you decide to undertake a similar effort.

In this blog, I'll show you how to set up a GitHub Actions workflow to automatically publish a new release of a GitHub repo to npm - no muss, no fuss, no manual input required.

Notehub JS

Ok, before we get to the GitHub Actions workflow, I want to give you a little background on the Notehub JS project because it's a bit unusual.

As I mentioned above, Notehub JS is a JavaScript-based library for interacting with the native Notehub API, and it's actually generated from the Notehub project's own openapi.yaml file, which follows the OpenAPI specification standards.

OpenAPI Specification

The OpenAPI Specification (OAS) defines a standard, language-agnostic interface to HTTP APIs which allows both humans and computers to discover and understand the capabilities of the service without access to source code, documentation, or through network traffic inspection. When properly defined, a consumer can understand and interact with the remote service with a minimal amount of implementation logic.

An OpenAPI definition can then be used by documentation generation tools to display the API, code generation tools to generate servers and clients in various programming languages, testing tools, and many other use cases.

OpenAPI Generator CLI

Because the Notehub API has an openapi.yaml file that is updated regularly, I was able to use the OpenAPI Generator CLI to automatically generate a JavaScript-based library to interact with the Notehub API in just a few commands from a terminal.

The catch is, when the new folder is generated which has all the API endpoints and models, the documents, and the dist/ folder that packages everything up for consumption as an npm module, it's stored as a subfolder inside of the main Notehub JS repo.

The openapi.yaml file lives at the root level of the project along with its config.json file, a package.json file, and a few other other bits and bobs (license, contribution guidelines, code of conduct, etc.), but the real meat of the library lives inside of the src/ subfolder.

Here's a simplified view of the Notehub JS repo's folder structure:

root/
├── .github/ 
| ├── workflows/
| | ├── publish-npm.yml 
├── src/ <- this is the folder generated by the OpenAPI Generator CLI
| ├── dist/
| ├── docs/
| ├── src/ 
| | ├── api/ 
| | ├── model/
| | ├── index.js 
| openapi.yaml
| config.json
| package.json

From the diagram above you can see the openapi.yaml file that this library is generated from lives at the root of the repo, and the src/ folder is what actually holds all the Notehub API JavaScript code that powers the Notehub JS library.

This src/ folder is what needs to be published to npm - not the root of the repo as is the case with most projects in GitHub. But not to worry, publishing just this subfolder can be done, and better yet, it can be automated.

Configure the root `package.json` to point to the subfolder where the Notehub JS library code lives

So now that I've covered why I only need to publish the subfolder of the Notehub JS repo to npm, let's get on to the how.

First thing: add a few extra configuration details to the package.json at the root of the project.

Update the main field to point to the subfolder's index.js file

According to the npm docs, the main field points to the primary entry point of an app, and when it's not specifically defined it defaults to index.js at the root of the project.

Since the src/dist/ folder is where the index.js file lives for Notehub JS library code that should be uploaded to npm, main needs to be updated to point to it instead.

Now the package.json's main field should read:

"main": "./src/dist/index.js"

Update repository to include a directory field

Next, the repository field in a package.json specifies where the code lives, and it's generally helpful for people who want to contribute to the project.

It has another benefit though: if the package.json for the npm package is not in the root directory (like for us, where the auto-generated src/ folder has its own package.json), you can specify the directory in which that file lives.

Here's what the repository field should look like:

"repository": {
  "type": "git",
  "url": "https://github.com/blues/notehub-js.git",
  "directory": "src"
}

What this translates to on the npm package page is a handy link that points back to root of the GitHub folder (see image below).

</figure>

Add the publishConfig field to specify directory

Typically, the publishConfig field is used to set config values that will be used at publish time like tags, registries, or access.

Another field that can be added here is directory, which customizes the published subdirectory relative to the current package.json. This ensures npm knows the package.json it needs to read from (and display) in the npm package page is inside of the auto-generated src/ folder, not at the root of the project.

Add the following publishConfig code to the package.json like so:

"publishConfig": {
  "registry": "https://registry.npmjs.org",
  "directory": "./src"
}

In the end, the whole package.json at the root of the Notehub JS repo looks like the code snippet below.

NOTE: I've omitted parts of the package.json for clarity, but if you'd like to see the whole file, you can click the file name below. It's linked to the actual file in the repo in GitHub.

package.json

{
  "name": "notehub-js",
  "description": "JavaScript library for accessing the Blues Wireless Notehub API endpoints",
  "main": "./src/dist/index.js",
  "repository": {
    "type": "git",
    "url": "https://github.com/blues/notehub-js.git",
    "directory": "src"
  },
  "publishConfig": {
    "registry": "https://registry.npmjs.org",
    "directory": "./src"
  }
}

Great, the changes to the package.json to publish just the src/ folder to npm are complete, let's move on to the GitHub Actions workflow to make publishing a new version of the Notehub JS library to npm as automated as possible.

Create a GitHub Actions workflow to publish to npm

Need a refresher on GitHub Actions?

If you want a quick primer on what GitHub Actions are, I recommend you check out a previous article I wrote about them here.

I decided that for my use case, creating a new release made the most sense to trigger a GitHub Actions workflow to publish the Notehub JS src/ subfolder to npm.

GitHub specifically defines a release as a deployable software iteration that you can package and make available for a wider audience to download and use, which is exactly what I want.

Once I had settled on this as the trigger for my GitHub Actions workflow, it was a pretty straightforward set of steps to publish to npm.

Here's what the finished publish-npm.yml file will look like inside of the ./github/workflows/ folder - I'll break it all down afterwards.

publish-npm.yml

name: Publish notehub-js to npm

on:
  release:
    types: [created]

jobs:
  npm:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: ./src
    steps:
      - name: Check out code
        uses: actions/checkout@v3

      - name: Setup .npmrc file to publish to npm
        uses: actions/setup-node@v3
        with:
          node-version: "16.x"
          registry-url: "https://registry.npmjs.org"

      - name: Install dependencies
        run: npm ci

      - name: Publish to npm
        run: npm publish
        env:
          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

Each workflow file needs a name, so I chose a descriptive one: Publish notehub-js to npm.

Next, as I already explained above, this workflow is triggered whenever a new release is created in GitHub, which is where

on:
  release: 
    types: [created]

comes into play.

on is how a workflow is triggered.
release is the event that triggers the workflow.
types: [created] is the activity type for a release event that triggers the workflow. This gives us more fine-grained control of when the workflow should run.

Then the jobs section runs inside of the workflow. This particular script only has one job, npm, but if there's multiple jobs, they'll run sequentially unless otherwise specified.

The npm job defines that it runs on the latest version of Ubuntu in runs-on.

defaults.run is an important part of the script because it's where I specify that the working directory for all the following steps is the src/ folder (because I don't need any of the files from the root of the repo, just what's inside the src/ folder, I can define this at the job level).

Finally, I get to the steps.

The steps are as follows:

Check out the code so the workflow can access it using the GitHub Action actions/checkout@v3.
Download a Node.js version to build the package before publishing it to npm using the GitHub Action actions/setup-node@v3 (specify the node-version and registry-url for npm here).
Install the project's dependencies stored in the src/ folder's package.json file using npm ci.
Publish to npm by running the command npm publish and supplying it with an NPM_TOKEN (generated in npm) to verify this workflow has permissions to publish to the specified npm package registry.

And there you have it: each time a new release is created in the Notehub JS repo, this GitHub Actions workflow will run and deploy the updated code to npm.

Conclusion

Working as an engineer for a startup company is cool because no two days are alike for me. One day I'm building web apps and updating documentation on our developer experience site, the next I'm reading up on best practices for npm packages and how to automate tasks with GitHub Actions.

Venturing into the realm of open source software and publishing my first JavaScript package to npm was quite a learning experience for me, but I'm so grateful I had the chance to push myself into this space and create something useful for my team (and hopefully our users as well!).

One thing I knew I wanted, right from the start, was to automate the tasks I knew I'd be repeating fairly regularly over time: namely, regenerating the JavaScript library as the Notehub API continued to evolve, and redeploying a new version of the library to npm with the latest updates.

With some extra configurations in the Notehub JS repo's package.json, I was able to focus in on the auto-generated subfolder that contains the useful JavaScript-based code for easily interacting with the Notehub API, and with a few GitHub Actions inside of a workflow, I was able to build the code for the package and publish it to npm with the click of a button. Not too bad for an end result.

Thanks for reading. I hope learning how to target a specific folder inside of a project and automating deployments to npm through GitHub Actions workflows come in handy for you in the future. Enjoy!

References & Further Resources

Notehub JS GitHub repo
notehub-js npm library
OpenAPI Generator CLI

Tips and Tricks for Using the Prisma ORM

Wed, 22 Feb 2023 00:00:00 GMT

</figure>

Introduction

Recently, my team and I were building an application that monitored the flow rate of liquid through pipes and remotely controlled valves to enable or disable these flows.

I work for the Internet of Things company Blues which specializes in getting IoT data from devices in the real world into the cloud via cellular, and we were demonstrating how our tech could be used by facilities managers to monitor liquid and gas transfers and remotely toggle valves throughout a piping system via a web app.

Without going into too much detail, the way the project works is: the device sends the current flow rate data and valve state to the cloud, the cloud then forwards that information to our web application, and the application saves that data into a PostgreSQL database and displays it in the browser for the user. From the UI, the user can monitor the flow rate, checking for anomalies, and open or close a valve at will.

If you're interested to learn more about our flow rate monitoring project, you can see the full tutorial for building your own here.

The web app dashboard looks like this:

For the web app, we were using Next.js to display our IoT data dashboard, and Prisma (an object relational mapper or ORM) to interact with our PostgreSQL database. Because of the project requirements, which I'll get into shortly, we ended up doing some pretty interesting things with Prisma including upsert transactions, querying for the most recent record in a table, fetching related data from multiple tables in one query, handling raw JSON data, and more.

Today, I'm going to share with you some of the more advanced and complex solutions I learned to solve with Prisma.io in hopes of making it easier when you run into similar scenarios in your own application development.

NOTE: This article is not an introduction to working with Prisma. If you're new to it, I highly recommend checking out their getting started documentation first to get familiar.

Prisma primer

Rarely do web developers today write raw SQL queries in their applications. Instead, we tend to rely on object relational mapping libraries (ORMs) like Mongoose, Sequelize, Knex, or Prisma.

ORMs provide a host of benefits like:

Time saved writing raw queries,
Compatibility with multiple types of databases,
Schema definitions,
Type-safety,
And more.

Prisma, in particular, has become the latest, open source favorite when it comes to ORMs because it provides all the benefits I mentioned above plus:

VS Code integration,
TypeScript type-safety,
Pagination and transactions,
Serverless support,
And a visual database browser, to name a just few.

It is really slick to work with and makes interacting with a database in a JavaScript application feel much more intuitive and natural and much less like two different languages being mashed together.

But even with all this going for it, there are still some things that are just plain complex to do, so let's cover some of them now so they're less daunting when you run into them in your next project.

Define implicit many-to-many data relationships

The majority of the tips I want to share have to do with tables of related data, so the first thing we'll cover is how to define those type of relationships.

For our project, we had two Prisma models that were related to one another:

Device - each individual flow rate and valve control device,
Fleet - a particular group of these devices.

NOTE: For any of the code snippets listed below, click the titles of the snippets to see the working code in GitHub repo.

The Device model in Prisma schema looks like this:

Device model

model Device {
  id            Int         @id @default(autoincrement())

  // unique device uid in notehub
  deviceID     String      @unique @map("device_id") @db.VarChar(64)

  // device serial number (cached)
  name          String?     @db.VarChar(80)
  lastSeenAt    DateTime?   @map("last_seen_at") // the last time an event was heard from this device

  // the project the device belongs to
  project       Project     @relation(fields: [projectID], references: [projectID])
  projectID    String

  @@map("device")
}

The Fleet model in the Prisma schema looks like this:

Fleet model

model Fleet {
  id          Int      @id @default(autoincrement())

  // unique fleet id
  fleeID    String   @unique

  // the project the fleet belongs to
  project     Project   @relation(fields: [projectID], references: [projectID])
  projectID  String

  @@map("fleet")
}

Each device can belong to no fleets or multiple fleets, and each fleet can contain no devices or many devices. This is a classic example of a many-to-many (m:n) relationship.

Prisma offers two ways to handle these sorts of relationships:

Explicit many-to-many relationships - in explicit m:n relationships, the relation table is represented as its own model in the Prisma schema and can be used in queries. Explicit many-to-many relations define three models: Two models that have a many-to-many relation: Device and Fleet, and one model that represents the relation table: DevicesInFleets (sometimes called the JOIN, link or pivot table) in the underlying database.
Implicit many-to-many relationships - in implicit m:n relationships, the table exists in the underlying database, but it is managed by Prisma and does not manifest in the Prisma schema (i.e. you don't have to define the DevicesInFleets model and table yourself, Prisma just knows it's a thing).

Here is a typical JSON Device data event that might get routed to the web app.

Example event routed to the app

{
  "projectID": "app:a8beb5bd-622e-46a6-866a-ae4528c7c201",
  "deviceID": "dev:864622040363787",
  "eventID": "490006f7-80f8-4314-8b5e-5ff587f07fba",
  "lastSeenAt": "2023-01-23T18:29:16.000Z",
  "eventBody": { "flow_rate": 810, "valve_state": "open" },
  "fleetIDs": [ "fleet:f660d491-8830-420f-be7a-c8f91c460813" ],
  "name": "Pipe Section A"
}

For convenience, we chose to take advantage of Prisma's implicit many-to-many relationship option to associate devices and fleets with one another. In order to let Prisma know that devices and fleets are related , in the Device and Fleet models defined in the schema.prisma file, simply add each related model to the other's schema.

So the Device model has Fleet added to it:

Device model

model Device {
  id            Int         @id @default(autoincrement())

  // unique device uid in notehub
  deviceID     String      @unique @map("device_id") @db.VarChar(64)

  // extra device-specific data here

  // fleets a device belongs to (implicit many-to-many relationship)
  fleets        Fleet[]

  @@map("device")
}

And the Fleet model has Device added to it:

Fleet model

model Fleet {
  id          Int      @id @default(autoincrement())

  // unique fleet id
  fleeID    String   @unique

  // more fleet-specific data

  // devices assigned to fleet (implicit many-to-many relationship)
  devices     Device[]

  @@map("fleet")
}

And that's all you need to do to establish an implicit many-to-many relationship with Prisma. No explicit tables, no SQL queries, no extra work to handle a relatively complex, yet very commonplace, relationship. I can tell you, implicitly defining these sorts of relationships amongst a number of related tables in this project really saved us a ton of time during development.

Now that we've defined the related data, it's time to handle creating or updating related data in one function.

Accurately update related records across multiple tables with Prisma

Building off the previous example, we'll cover here how to correctly update related data across multiple tables at once.

As I said before, each device can belong to zero or more fleets, and each fleet can contain zero or more devices at any given time.

As new Device events flow into our PostgreS database from individual devices, each event includes a field with a list of fleet IDs for whichever fleets that device belongs to at the time (if any).

Example event routed to the app

{
  "projectID": "app:a8beb5bd-622e-46a6-866a-ae4528c7c201",
  "deviceID": "dev:864622040363787",
  "eventID": "490006f7-80f8-4314-8b5e-5ff587f07fba",
  "lastSeenAt": "2023-01-23T18:29:16.000Z",
  "eventBody": { "flow_rate": 810, "valve_state": "open" },
  "fleetIDs": [ "fleet:f660d491-8830-420f-be7a-c8f91c460813" ],
  "name": "Pipe Section A"
}

According to Prisma's docs, the connectOrCreate API function is the way to create a "related record" that may or may not exist, which is perfect for adding new devices to the Device table and new, related fleets to the Fleet table at the same time via a create() with nested write function. However, when attempting to update a device's fleets when a new event came in, I encountered a slight problem with the update() function.

When an event arrives for an existing device and the fleets are completely different, only having the connectOrCreate() function present in the update() failed to delete the old fleets and replace them with new fleets in the Fleet table. The new fleets were instead associated with the device along with the old fleets, and the device looked as if it was associated with more fleets than it truly was.

In order to accurately update both the Device table and the related records in the Fleet table in the same query, we have to do two things:

Disconnect all related fleet records using the set() method.
Call the connectOrCreate() function to insert the new fleet data into the Fleet table.

NOTE: Deleting specific related records with deleteMany() function does not work

Another option for altering related records that I tried was to delete all related records using the deleteMany() function. This didn't work correctly though because it deleted the fleet IDs from all the devices in the Device table, not just the deviceID the event specified.

Here is what the final upsertDevice() query looks like: it correctly updates the device data in the Device table, and the associated fleet IDs in the Fleet table without altering the fleets associated with any other device in the process.

upsertDevice() query

 /**
   * Insert or update the device based on the unique device ID.
   *
   * @param project
   * @param deviceID
   * @param name
   * @param lastSeenAt
   * @param fleetIDs
   * @param location
   * @returns
   */
  private upsertDevice(
    project: Project,
    deviceID: string,
    name: string | undefined,
    lastSeenAt: Date,
    fleetIDs: string[],
    location?: NotehubLocation
  ) {
    const args = arguments;

    const formatConnectedFleetData = fleetIDs.map((fleet) => ({
      create: {
        fleetID: fleet,
        projectID: project.projectID,
      },
      where: {
        fleetID: fleet,
      },
    }));

    return this.prisma.device
      .upsert({
        where: {
          deviceID,
        },
        create: {
          name,
          deviceID,
          locationName,
          fleets: {
            connectOrCreate: formatConnectedFleetData,
          },
          project: {
            connect: {
              id: project.id,
            },
          },
          lastSeenAt,
        },
        update: {
          name,
          locationName,
          fleets: {
            set: [],
            connectOrCreate: formatConnectedFleetData,
          },
          project: {
            connect: {
              id: project.id,
            },
          },
          lastSeenAt,
        },
      })
      .catch((cause) => {
        throw new ErrorWithCause(
          `error upserting device ${deviceID} ${JSON.stringify(args)}`,
          { cause }
        );
      });
  }

Be aware that the set() function does not actually delete the fleet ID from the Fleet table - it just disconnects it from the device, however, as long as you don't have large amounts of fleet ID records filling up your related table, this shouldn't pose any major performance problems for your app.

Unfortunately there's currently no single related record upsert() function that will both delete previously related records and create newly related records, but maybe one day Prisma will include this use case. Until then, I hope this helps you handle accurately updating your related records.

Ok, let's move on to the next complex Prisma tip: including relational data in a READ query.

Return relational data from another table in Prisma READ query

Just like updating related tables can be done in a single transaction with Prisma, querying multiple tables for related data can also be done in a single action.

Taking the web app we've been talking about, for display purposes in the UI, it's necessary to have both the individual device ID and the fleet IDs that each device belongs to. If you've been following along with this article, you'll know already that the device details reside in the Device table and the fleet details for fleets that device is part of reside in their own separate Fleet table.

When storing device and related fleet data in their proper tables, we can use the connectOrCreate API function.

To perform a "nested read" and fetch both the device details and the associated fleet details, we use Prisma's include API function.

To get all the fleet data associated with a particular device, the fetchDevice() query in our code base looks like this:

fetchDevice() function

async fetchDevice(deviceID: DeviceID) {
    const device = await this.prisma.device.findUnique({
      where: {
        deviceID: deviceID,
      },
      include: {
        fleets: true,
      },
    });
    return device;
  }

We use Prisma's standard findUnique API query on the Device table to find the specific device based on its device ID, and then inside of that findUnique() query, we add include: { fleets: true } to get all the fleet details that are related to that device.

Here is what the data that comes back from the query looks like:

Example device data with fleet info returned from query

{
  [
   {
      "id":1,
      "deviceID":"dev:864622040363787",
      "name":"Pipe Section A",
      "lastSeenAt":"2023-01-23T18:29:16.000Z",
      "projectUID":"app:a8beb5bd-622e-46a6-866a-ae4528c7c201",
      "fleets":[
         {
            "id":1,
            "fleetID":"fleet:f660d491-8830-420f-be7a-c8f91c460813",
            "projectID":"app:a8beb5bd-622e-46a6-866a-ae4528c7c201"
         }
      ]
   }
 ]
}

As you can see from the code above, all the relevant device info is provided along with the fleet data for its related fleet. Pretty sweet.

NOTE:

If you only wanted to return certain fields for related data, you could use the select API to choose a subset of related fields to return.

For example, if you only wanted related fleet IDs, you'd update the include query to be: include: { fleets: { select: { fleetID: true } } } }.

Get most recent data record from a table

Right, let's move on to reading the most recent data record from a table.

In the UI of the valve flow rate monitor, we need to get the most recently added flow-rate and valve state data for a device to correctly show the user what's happening.

Prisma provides a findFirst API query that returns the first created data record, but it doesn't provide a similarly named findLast API query.

Instead, there are two ways we can get the most recent record out of a table:

Use a combination of the findFirst API and the orderBy filter to sort the records by something like the record's date.
Use the findMany API and use the take function to grab just the last value from the query.

We'll cover both options:

findFirst and orderBy example

The first way to get the most recent event from a table is to use findFirst combined with orderBy where the records are sorted in descending order from newest to oldest. In the app, there are alarms that are triggered when a device's flow rate falls below or above thresholds set by the user; the alarm is displayed by highlighting the device's row in red in a table showing all the devices.

Since the device row can't be highlighted more than once, we don't need to know if the device has more than one alarm event for it - we only need to know the most recent alarm event, so we'll use the findFirst query and sort all the alarm events in the Notification table for a device in descending order by their lastSeenAt date.

Here's how the final query looks.

getLatestDeviceAlarm()

  async getLatestDeviceAlarm(deviceID: DeviceID): Promise<Notification> {
    const latestAlarmFromDevice = await this.prisma.notification.findFirst({
      where: {
        AND: {
          type: "alarm",
        },
        content: {
          path: ["deviceID"],
          equals: deviceID,
        },
      },
      orderBy: {
        lastSeenAt: "desc",
      },
    });

    return latestAlarmFromDevice || undefined;
  }

Notice in the query that in the end, the getLatestDeviceAlarm() function either returns the most recent alarm for a device or it returns undefined because there's a chance the device has not had an alarm go off. Something to keep in mind, which I'll cover in more detail later in this article.

Now let's look at the other option to get the most recent record from the Event table using findMany.

findMany and take example

getLatestDeviceEvent()

async getLatestDeviceEvent(deviceID: DeviceID, file: string): Promise<Event> {
    const latestDeviceEvent = await this.prisma.event.findMany({
      where: {
        AND: {
          deviceUID: deviceID,
        },
        eventName: file,
      },
      take: -1,
    });
    return latestDeviceEvent[0];
  }

For this function, we just want to get the latest regular event for a device: the latest flow rate and valve state reading - again, it's used to display the device's current status in the web app dashboard.

To get the most recent event with Prisma's findMany API, we use the deviceID to narrow down all the events and use take: -1 to grab the last record in the table. take: -1 was introduced to Prisma back in v2 to simplify paginating through lots of data, but it can also be used just to return one record as well as a list of records, just remember to extract the single record from the array when accessing the data.

Read and write JSON data to a table

All right, now we'll cover a very useful and flexible thing you might want to do: write JSON data into a table column or read it out again.

JSON fields are super useful when you need to store data that doesn't have a consistent structure (i.e. you're not sure what properties will be present or not), or you're importing data from another system and don't want to map that data to Prisma models.

For our project, we chose to use JSON fields for event objects because we weren't always certain what data an event might contain. Here's what an event routed to the app might look like, the eventBody property is what we want to store as raw JSON data because it's the field most subject to change.

Example event routed to the app

{
  "projectID": "app:a8beb5bd-622e-46a6-866a-ae4528c7c201",
  "deviceID": "dev:864622040363787",
  "eventID": "490006f7-80f8-4314-8b5e-5ff587f07fba",
  "lastSeenAt": "2023-01-23T18:29:16.000Z",
  "eventBody": { "flow_rate": 810, "valve_state": "open" },
  "fleetIDs": [ "fleet:f660d491-8830-420f-be7a-c8f91c460813" ],
  "name": "Pipe Section A"
}

To get our data table prepared to accept any old JSON data as a column, we just need to tell the model that's the case. So in our schema.prisma file where we define all our Prisma models, we set the properties for an Event like so:

Event model

model Event {
  id           Int         @id @default(autoincrement())

  eventName    String      @map("file")
  eventID     String      @map("event") @unique

// the device that produced the event
  device       Device      @relation(fields: [deviceID], references: [deviceID], onDelete: Cascade)
  deviceID    String 

// when the event occurred
  when         DateTime
  value        Json

  @@map("event")
}

The property to pay special attention to is the value property - that's the field we're giving a type of Json to, which is all we need to do to make the model happy.

Write JSON into a Prisma field

After the model's defined, when a new event comes in (like the example above), a Prisma upsert() function can be called on the Event table.

upsertEvent() function

  /**
   * Insert or update the event
   *
   * @param deviceID
   * @param when
   * @param eventName
   * @param eventID
   * @param value
   * @returns
   */
  private upsertEvent(
    deviceID: string,
    when: Date,
    eventName: string,
    eventUID: string,
    value: object
  ) {
    const args = arguments;

    return this.prisma.event
      .upsert({
        where: {
          eventID,
        },
        create: {
          deviceID,
          when,
          eventName,
          eventID,
          value,
        },
        update: {
          // reading already exists
          // no-op
        },
      })
      .catch((cause) => {
        throw new ErrorWithCause(
          `error upserting event ${deviceUID} ${JSON.stringify(args)}`,
          { cause }
        );
      });
  }

If you're using TypeScript like we are in our project, just define the JSON field value as an object and then simply pass it in as one of the values in the Prisma upsert() function. Pretty straightforward.

Read JSON from a Prisma table

Reading the JSON field out with Prisma is pretty straightforward as well. If you recall in the previous tip we looked at the getLatestDeviceEvent() query, this query returns the data from the Event table, including the JSON field value. When the data is returned it looks something like this:

Example event data returned from getLatestDeviceEvent()

[
  {
    "id": 1,
    "eventName": "data.qo",
    "eventID": "490006f7-80f8-4314-8b5e-5ff587f07fbc",
    "deviceID": "dev:864622040363787",
    "when": "2023-01-23T18:29:16.000Z",
    "value": { "flow_rate": 810, "valve_state": "open" }
  }
]

From here, just like with any other field of data returned by a Prisma query, we can access the value field (and any data contained therein) and do with it what we want. So to get the flow_rate, you'd simply do something like const flowRate = data[0].value.flow_rate;.

Trust me, this ability to easily read and write any sort of JSON data into a field in a database via a Prisma model is really, really handy to have.

Handle undefined data

Last Prisma tip for this article: be prepared to handle undefined values returned from Prisma.

As I alluded to in the earlier tip where I showed the getLatestDeviceAlarm() query, there is a chance in some scenarios where there are no records that match the parameters in the Prisma query. When this happens, undefined will be returned by Prisma and if you're not prepared to handle it, it can throw you for a loop.

If you're calling a particular Prisma query for an array of items (like checking which devices have alarm events), be ready to filter out any returned alarm data that is actually undefined so it doesn't give you false positives.

And with that, we've covered the majority of the hard won knowledge I gained about Prisma while working with it.

Conclusion

Prisma.io is a very powerful, very popular, JavaScript-friendly ORM for interacting with all sorts of databases. Although the learning curve for it can be a little steep, once you start to get the hang of it, you realize just how fine-grained the control is and how much nicer it is to use when interacting with databases instead of raw SQL queries.

When my team and I were using it to help us build a valve and flow rate monitoring application, we learned a lot about using some of Prisma's more advanced features like relational data tables, handling raw JSON fields, and even filtering and sorting results. With Prisma, very quickly you can start to do some amazing stuff that used to be quite complex and required good SQL knowledge.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading. I hope you learned some new ways to use Prisma on top of your own database-driven web apps. Enjoy!

References & Further Resources

Prisma.io ORM site
Valve flow rate monitor Hackster tutorial
Valve flow rate monitor GitHub repo
Many-to-many (m:n) data relationship

Use CSS to Reveal a Child Element in the DOM by Interacting with the Parent

Sun, 18 Dec 2022 00:00:00 GMT

import YouTube from '../../src/components/YouTube.astro';

</figure>

Introduction

Building my own personal site was a lot of fun because I got to spend time on the little details and subtle touches to give it that little something extra - I know I always appreciate when I see them on other sites. One such detail that I wanted to add was a little arrow that appears right next to the "Read more" link on cards for each of my blog posts when a user hovers over it.

<figcaption>See this little arrow next to read more? It's only there when a user mouses over the card.</figcaption>

</figure>

At first glance, it sounds like a relatively simple matter for the CSS :hover selector, but what I actually wanted to happen was when a user hovered over any part of my card, the arrow would appear - not just the "Read more" link in the bottom left corner of the card. What I really wanted was for the larger parent container to trigger a CSS change in a child element. Getting this to work proved a little trickier to figure out how. But I persevered, and in the end, had a little arrow that appears and disappears depending on a user's mouse. It's an understated addition and I like it - almost like an Easter egg.

This article will show you how to use an interaction that a user takes on a parent element (like hovering over it) in the DOM to make a change in a child element via CSS. (I can see lots of situations where something like this could come in pretty handy.)

Below is a video showing how the arrow next to the "Read more" text on the card appears no matter where a user mouses within the card: title, body, tags - anywhere in the card makes the arrow appear, not just when a user mouses over the link itself.

Make a blog post card's HTML (or JSX)

</figure>

Before we get to the CSS portion of this post, we first need to set up the general outline of a blog post card's HTML.

My site is built with React using the Gatsby framework. In my code examples, you'll see a smidge of React and JSX instead of traditional HTML, but the CSS is really where our focus will be for the majority of this article so don't worry too much about it.

NOTE: I've omitted most of the React-specific code like Hooks, state, functions, etc. to better focus the scope of this article, but if you want to see the full code, you can click the title of any of the code examples to see my full repo files in GitHub.

Here is a simplified look at the LatestPosts component which fetches all the blog posts available on my site, and takes the most recent four to display in a list of cards.

LatestPosts.js

// other imports for React, components, constants, etc.
import './LatestPosts.scss';

const LatestPosts = () => {
  // state variables here

  // logic to fetch all blog posts, take the most recent 4 posts and format the data to display

  return (
    <div className="latest-posts-wrapper">
      {posts.length
        ? posts.map((post, index) => (
            <div className="post" key={index}>
                <Link
                  className="post-link"
                  to={`/blog${post.path}`}
                  key={post.title}
                >
                  <p className="post-title">{post.title}</p>
                  <p className="post-sub-title">{post.subTitle}</p>
                  <PostTags tags={post.tags} />
                  <p className="read-more">
                    Read more <span className="arrow">&gt;</span>
                  </p>
                </Link>
            </div>
          ))
        : null}
      <h2>
        <Link to="/blog">View all articles &gt;</Link>
      </h2>
    </div>
  )
};

export default LatestPosts;

In this component code, there's a single <div> (as required by all React components) that contains all the posts displayed in this list: it's the <div> with the latest-posts-wrapper class.

Then, the list of posts is iterated over with .map() and each individual post is rendered as an element in the DOM that looks like a card. Each card has a link to the post, the blog title, subtitle, tags, and the text Read more encouraging users to click through.

Take note that the <Link> element wraps the majority of the blog post content - that will be key soon enough.

Inside of the final <p> tag with the class of read-more is a <span> tag that surrounds the > - this is the HTML for the greater than sign (>) which I'm using as a right arrow symbol.

Finally, there's another header tag at the bottom of the component, <h2>, inviting users to go the site page with all the blog posts.

Ok, now that the HTML groundwork's been laid, let's get on with the CSS and making these arrows appear appear and disappear as required.

Add the CSS to the component

Now there's a bit more SCSS associated with the <LatestPosts> component, but for clarity I've omitted the bits not relevant to the blog. If you'd like to see the unabridged version of the code, you can click the name of the file and see it in GitHub.

NOTE: The code below is actually SCSS instead of pure CSS because I greatly appreciate the additional features like class nesting that it provides, but it should be pretty simple for you to translate this as needed.

Ok, here's the CSS: now let's talk about what's happening.

LatestPosts.scss

.latest-posts-wrapper {
  display: flex;
  flex-direction: column;

  .post {
    margin: 10px 20px;
    padding: 20px 30px;
    background-color: white;
    box-shadow: rgba(0, 0, 0, 0.1) 0px 1px 3px 0px;
    border-radius: 5px;

    .post-link {
      // post-related css

      .arrow {
        opacity: 0;
      }
    }

    .post-link:hover {
      // more post-related css

      .arrow {
        opacity: 1;
      }
    }
  }
}

As you look at this code, take special note of the classes .post-link and .arrow - they're the keys here.

.post-link is the class that surrounds all the content within each card. arrow is the class that goes with the greater than arrow.

In order to keep the arrow hidden from view until a user hovers over the card, we'll set the arrow class to opacity: 0; inside of the post-link class. Even though it's still there in the HTML, it's not visible onscreen.

Then, when a user hovers over a card, we actually invoke the CSS pseudo class :hover on the post-link class and switch the arrow class' opacity to opacity: 1;, and that makes the arrow show up as soon as a mouse enters anywhere inside the card.

Instead of adding the :hover to the read-more class that would only make the arrow show when a user mouses over the "Read more" link, we add the :hover higher up in the card element which makes the arrow show up almost anywhere the user's mouse is on the card. Voila!

And that's all there is to it: simply shifting the :hover pseudo class that surrounds the child element further up in the larger parent component can make it happen.

Once more, here's what the finished product looks like: hover almost anywhere on the card the arrow appears, mouse off of it, the arrow disappears again.

Conclusion

One of the coolest parts of being a web developer is bringing things to life online, and even though I am not a designer, it was really fun to flex my design muscles as I built my own site, and add all sorts of little touches that I liked.

Things like gradients here and there, subtle animations, and the like: nothing too dramatic mind you, but small things that I always appreciate when I see it on other sites.

Not too long ago, a touch I wanted to add was when a user moused over one of my blog cards they'd see a little arrow next to the "Read more" prompt pop up. But I wanted the arrow to pop up anywhere a mouse might hover over the card - not just when they got the "Read more" link in the bottom of the card.

By employing the :hover pseudo-class on one of the card's classes that wraps all of the card's content (and not just the "Read more" link), I was able to achieve this effect and it didn't take very much effort at all. Pretty sweet.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading. I hope this trick to affect a child element's CSS by interacting with a parent or grandparent element comes in as handy for you as it does for me.

References & Further Resources

Create Upsert Yourself for Amazon Redshift Databases

Sat, 19 Nov 2022 00:00:00 GMT

</figure>

Introduction

One of the many things I enjoy about working at a startup is the variety of people I get to work with across the company and the interesting problems I get to solve. A few months back, the marketing team was working on improving our analytics reporting and wanted to store data in our Amazon Redshift database that could then be used to build more detailed reports.

While this would normally be a relatively straightforward INSERT SQL request if all the data was brand new, one thing that made it a little tricky was that I needed to update certain rows in the data table AND insert new rows as well. Normally, I'd use an UPSERT SQL statement to do both things at once, but Redshift doesn't support UPSERT statements, so I had to get a little more creative to make this happen manually. With some research and a lot of testing, I learned that by using a temporary data table and a few SQL statements strung together it was possible, and it wasn't as complicated as I thought it would be.

Let's look at how to create our own version of an UPSERT SQL statement in a PostgreSQL-based Amazon Redshift database, updating data that already exists and adding new data as well.

Amazon Redshift

Redshift is Amazon Web Service's PostgreSQL-based cloud data warehouse built with added functionality to manage very large datasets and support high-performance analysis and reporting of those data.

It's similar to Google's BigQuery or Microsoft's Azure cloud data warehouses, if you've ever used either of those in the past.

And although Redshift is based on PostgreSQL it does have some differences, one of those being: a lack of UPSERT functionality.

No built in UPSERT functionality

Amazon acknowledges this shortcoming readily enough, and suggests using a staging table to perform a merge. This sounds sensible, but the AWS documentation on how exactly to do this is a little sparse.

So after I figured it out, I wanted to share it, with code examples, for each step. Let's get to it.

Create your own UPSERT

NOTE: This tutorial assumes you're familiar with basic SQL syntax. If you wish to reference any of the commands in the article, I recommend checking the Redshift SQL command docs.

If there's no native UPSERT, we'll make our own. There's more than one way to recreate UPSERT but the option I chose to go with essentially deletes all the existing rows in the production table that match rows in the temporary table, then inserts all the data from the temporary table into the production table.

It sounds complex (because it is a bit), but it's not as bad once each step in the process is broken down.

For this article, I'll set up a table (USER_UPDATES) that's a copy of a fictitious USERS table. In USER_UPDATES, I'll modify some existing data in the table (updates), and create some new data in the table (inserts), then we'll get to updating that newly modified data into the USERS table.

Prerequisite data modification setup

-- Create a sample table as a copy of the fake USERS table 

CREATE TABLE user_updates AS
SELECT * FROM users;

-- Change every third row so we have updated users

UPDATE user_updates
SET 
    firstname = upper(firstname),
    lastname = upper(lastname),
    lastseen = '2022-11-13'
AND mod(userid, 3) = 0;;

-- Add some new rows of users so we have insert examples 
-- This example creates a duplicate of every fifth row of user data

INSERT INTO user_updates
SELECT 
      (userid + 127) AS userid, 
      firstname, 
      lastname, 
      getdate() AS lastseen,
FROM user_updates
AND mod(userid, 5) = 0;

Now that there's some updated user data, it's time to add it back to the USERS table, while ensuring the user data for users already in the table gets overwritten and updated, and the new user data gets added.

1. Make a staging table

The first thing we'll need to do is write a SQL command that creates a temporary table that is an exact copy of the production table the data will eventually be written into.

-- Create temporary table that's a duplicate of the production users table

CREATE TEMP TABLE temp_users (LIKE users);

Creating a temporary table is almost the same as creating a regular table in SQL (just add TEMP), and instead of having to list out each column in detail, we can simply use LIKE <table_name> to tell SQL to make this table's columns a copy of the table it will eventually add its data to.

2. Insert the new data into the temp table

With our temp table now created, it's time to insert the modified data in the prerequisite setup step into that temp table. In this INSERT statement, we'll take all the data from the USER_UPDATES table and insert it into the temporary TEMP_USERS table.

-- Take all the data from the user_updates table and insert it into the temporary table

INSERT INTO temp_users
SELECT
       userid,
       firstname,
       lastname,
       lastseen,
FROM user_updates;

Here, all the data from USER_UPDATES is put in the TEMP_USERS table for the moment.

3. Begin the transaction

A transaction is a unit of work done in a database - the work can be anything from creating tables to deleting them. The transaction taking place here, will be deleting particular rows of user data from the production table USERS (these are existing users who have updated data in the temp table) before adding their new data into the table.

-- Start the work of deleting existing user data from the table and adding the new data instead

BEGIN TRANSACTION;

The thing about transactions in PostgreSQL is that a commit is implicitly performed at the end of the statement as long as the execution is successful. If the execution fails, a rollback is done instead. This becomes especially convenient because if new user data fails to get added to the USERS table, it will rollback and automatically replace the old user data back in the table.

4. Delete the rows in the permanent table that need to be replaced with new data

Once the transaction's begun, the first thing to do is delete any rows from the permanent table (USERS) where the user ID matches a user ID in the TEMP_USERS table.

-- Delete the users from the database whose data needs to be updated

DELETE FROM users 
USING 
    temp_users 
WHERE users.userid = temp_users.userid;

By doing this DELETE first, we ensure that there will be no duplicate user data when the new data is added to the USERS table.

5. Insert all the data from the temp table to the permanent table

Now it's safe to insert everything from the TEMP_USERS table into the USERS table, without fear of duplicating existing user data.

-- Add all the user data from the temp table to the prod table 

INSERT INTO users
SELECT * FROM temp_users;

6. End the transaction

Once all the parts of the transaction have completed successfully, it can be ended just like it began.

-- End the work

END TRANSACTION;

There's one last thing to do now: a little clean up.

7. Delete the temp table

Finally, since the TEMP_USERS table has done its job and there's no further need for it, it's dropped (deleted).

-- Delete the temporary table now that its job is done

DROP TABLE temp_users;

Creating our own UPSERT's not so bad once each of the steps are broken down and explained.

Conclusion

Amazon Redshift is a popular, powerful data warehouse, but even though it's based on PostgreSQL, it lacks some of the nicer features like UPSERT, which is exactly the feature I needed to help out one of the teams at work.

Since I needed to update a data table with new information and update existing data in that table, I needed something more than a simple SQL INSERT. Lucky for me, creating UPSERT functionality wasn't as difficult as I thought it would be with the help of a temporary table and a couple of targeted DELETE and INSERT queries. Not too shabby.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading. I hope you find it helpful to see how to create your own SQL UPSERT function. This probably won't be the first or the last time you need to insert and update data into a table, and it's nice to have options for how best to accomplish it.

References & Further Resources

Rely on JavaScript's Intersection Observer to Execute Code When in View (or Not)

Sun, 23 Oct 2022 00:00:00 GMT

import YouTube from '../../src/components/YouTube.astro';

</figure>

It's the small things that matter

When dealing with the business of building websites on a daily basis, I can't help but start to notice that what separates the good sites from the great ones, is that the great ones pay close attention to the small details.

Little page animations, a button that disables after form submission, or when elements on the page only reveal themselves when the user scrolls to that portion of the page. You might not immediately recognize that this sort of nicety is there to improve the user experience, but it becomes more obvious when it's missing: a video that won't stop playing even after you've scrolled away, an error message that you can't see until you scroll up the page from the submit button that failed, there's a million little annoyances like this.

Recently, I was building a new About Us page for my company's marketing site, and in the mockup the designer gave me, there were some cool animations that showed a series of cards toward the bottom of the page that slid into view only when the user had scrolled far enough down the page to see them. It looked really slick, and it turns out that nowadays just a little CSS and JavaScript was all that was needed to make this possible, and a host of other handy interactions.

Today, I'll show you how the JavaScript Intersection Observer API can easily control how elements react based on their visibility in the viewport.

Below is a video showing how the cards at the bottom of this page don't animate and slide into view until the viewer has reached them: that's one example of Intersection Observer at work.

Intersection Observer

The Intersection Observer API, if you're not familiar with it, provides a way to asynchronously observe changes in the intersection of a target element with an ancestor element or a top-level document's viewport.

Put simply: it can tell if an element is in view or not, and cause things to act accordingly.

Before Intersection Observer existed this type of intersection detection was clunky with lots of event handlers and loops in the main thread, potentially causing performance problems as well as cluttered code.

An intersectionObserver, on the other hand, is created and assigned a callback function that runs whenever a threshold is crossed in one direction or the other for a particular element in the DOM that it is assigned to "observe."

It can also be given an optional series of options that tell it when to invoke the callback function:

The root - the element used as the viewport for checking visibility of the target.
The rootMargin - a set of values around the root element's bounding box to grow or shrink before computing intersections.
The threshold - a number that indicates at what percentage of the target's visibility the observer's callback function should be executed. (For example, the default 0 means as soon as even one pixel is visible the callback will run, and 1.0 means the threshold isn't considered passed until every pixel of an element is visible).

There's plenty more nuance to how the intersection observer can be fine tuned and leveraged to cool effect, but what I've covered above should be enough to help you follow the examples I'll be demoing. If you'd like to learn more, I'd recommend reading the docs from Mozilla, which include some great code examples to play with.

All of this writing may not make perfect sense yet, so let's look at some code examples where you can see intersection observers in action.

Intersection Observer code examples

There's a couple of different scenarios where intersection observer is being used on the marketing site in unique ways, and I'm going to show both of them to you to help demonstrate how flexible it can be.

Run animations only when the user will see the result

<figcaption>These cards slide into view once the user can see them on screen with the help of intersection observer.</figcaption>

</figure>

The first example I'll show you is the one I described in the introduction to this blog: it animates a series of cards on-screen, but only when the user has scrolled down the page far enough to view the cards.

NOTE:

The site code I'm referencing is written in Hugo, a popular, open source, static site generator written in Go. Like many SSGs, it relies on templates to render the majority of the site's HTML, and for the purposes of clarity in this article, I've replaced Go variables injected into the template with the generated HTML.

Let's look first at the HTML and JavaScript for the three cards being rendered in the page. This code snippet shows just one of the cards, but it's the same for all of them.

quote-cards.html

<div class="row">
    <div class="col-lg-4 card-group customer-quote-card">
      <div class="card">
        <div class="card-body">
          <p class="customer-quote-text">
            <img 
              class="customer-quote-img" 
              src="/images/about/quote-left-solid.svg" 
              alt="decorative left quotation mark"
            />
            "This is a quote here."
            <img
              class="customer-quote-img"
              src="/images/about/quote-right-solid.svg" 
              alt="decorative right quotation mark"
            />
          </p>
          <p class="customer-quote-source">
            Source of the quote
          </p>
          <p class="customer-quote-source-role">
            Role and company the source works for
          </p>
        </div>
      </div>
    </div>
  <script type="text/javascript">
    /* this code keeps the cards from animating into view until a user's scrolled down far enough to see them */
    var quoteCard = document.querySelectorAll(".customer-quote-card");

    var observerOptions = {
      threshold: 0.7,
    };

    var callback = (entries) => {
      entries.forEach((entry) => {
        if (entry.isIntersecting) {
          entry.target.classList.add("animated");
        }
      });
    };

    var quoteCardObserver = new IntersectionObserver(callback, observerOptions);

   // loop through each card available and add this observer to it
    quoteCard.forEach((card) => {
      quoteCardObserver.observe(card);
    });
  </script>
</div>

The HTML in the snippet outlines the card element and the details contained within that card: the quote, the name, and the company and role of the person being quoted.

The JavaScript at the bottom of the snippet is where intersection observer comes into play. First, a variable named quoteCard is created to target the customer-quote-card CSS class on the div that surrounds each card.

Then, the observerOptions variable is declared with a threshold of 0.7 - this will be passed to the IntersectionObserver object shortly, and will require that at least 70% of the card be visible before animating it into view.

A callback() function is defined, which loops through a list of items and for each item, if the item isIntersecting (a boolean value which is true if the target element intersects with the intersection observer's root), the animated class is added to that item's CSS classes. Essentially, this function will add the CSS class to each card necessary to trigger the animation and bring them into view on the page when the isIntersecting threshold is reached.

Next, a new IntersectionObserver instance is instantiated as quoteCardObserver and the callback() and observerOptions are passed in.

Finally, for each quoteCard object, the quoteCardObserver function is attached to the card.

With the HTML and JavaScript is set up, it's time to add the CSS (remember that animated CSS class mentioned earlier?) to animate the cards on screen when the intersection observer's callback() function fires.

about.scss

.card-group {
  opacity: 0;
}

.animated {
  animation: slideLeft 1.5s;
  animation-delay: 0s;
  animation-fill-mode: backwards;
  opacity: 1;
}

@keyframes slideLeft {
  0% {
    transform: translateX(100%);
    opacity: 0;
  }

  100% {
    transform: translateX(0);
    opacity: 1;
  }
}

In the SCSS code, the card-group class starts out with an opacity of 0 to keep the cards hidden from view.

The animated class is created with an animation property designating the slideLeft keyframe with a duration of 1.5 seconds.

In addition to the animation property, the animation-delay, animation-fill-mode and opacity are also defined on this class.

animation-delay: 0; ensures the animation will play immediately,
animation-fill-mode: backwards; means the element will apply the values defined in the 0% keyframe as soon as it is applied to the target (i.e. the card will stay hidden with an opacity of 0 as soon as it's rendered in the DOM).
opacity: 1; once the slideLeft keyframe animation has ended, the card will have 100% visibility on-screen.

And last but not least, the @keyframes animation sequence is defined. There are only two keyframes defined (0% and 100%) to indicate the cards start with no opacity (opacity: 0;) and 100% off the screen to the right (transform: translateX(100%);, and when it ends they'll be completely visible (opacity: 1;) and on screen (transform: translateX(0);).

In the end, it produces this effect:

NOTE:

CSS animations are beyond the scope of this tutorial, but if want to study them in more depth, I'd recommend starting here - they're very cool!

Great! That's one example of how intersection observer can be used to control animation timing so a user will see it. Now let's consider another option.

Play a video only when the user can see it

<figcaption>This video will only play when it is visible to the user and they click the play button. If user dismisses the modal by clicking somewhere else on the page, the modal will hide and video stop playing.</figcaption>

</figure>

The second example I'll share concerns a video modal that stops playing when the modal is not visible. When a button is clicked, the modal does a full page takeover where the video player sits at the center of the viewport. If the user clicks somewhere besides the video while it's playing, the modal is dismissed and hidden from view and the video will stop playing. This is possible because of intersection observer.

Let's take a look at the code that makes this happen.

video_modal.html

<div class="modal">
  <div class="modal-dialog">
    <div class="modal-content">
      <div class="modal-header">
        <button
          type="button"
          class="close"
          data-dismiss="modal"
          aria-label="Close"
        >
          <span aria-hidden="true">&times;</span>
        </button>
      </div>
      <div class="modal-body embed-responsive-16by9">
        <iframe
          width="560"
          height="315"
          src="https://www.youtube.com/embed/4q1qqzDzJeQ"
          title="YouTube video player"
          frameborder="0"
          allowfullscreen
        ></iframe>
      </div>
    </div>
  </div>
  <script type="text/javascript">
    var iframe = document.querySelector("iframe");
    var ytsrc = iframe.src;

    var observer = new IntersectionObserver(
      function (entries) {
        // isIntersecting is true when element and viewport are overlapping
        // isIntersecting is false when element and viewport don't overlap
        if (entries[0].isIntersecting === false) iframe.src = "";
        else iframe.src = ytsrc;
      }
    );

    observer.observe(iframe);
  </script>
</div>

In the code above, a video modal and an iframe video embed from YouTube are created in the HTML.

In the JavaScript directly below it, an iframe variable is defined focused on the <iframe> HTML element, and a ytsrc variable is defined to keep track of the the iframe's video source - this will be used inside the intersection observer's callback.

Next, a new intersection observer is initialized (observer), and this one's callback function iterates through the list of entries it receives, and if the first entry in the list is not intersecting (i.e. the video iframe and the viewport aren't overlapping, or rather, the modal's not visible to the viewer), the iframe's src is set to an empty string so no video URL is available to play. If the entry is intersecting (i.e. the video modal is visible in the viewport), the iframe's src is set to the ytsrc variable: the actual YouTube video URL.

Finally, the new observer object is told to observe the iframe variable, so whenever the iframe is in the viewport (e.g. when a user clicks a button to open the modal), its video source is the YouTube video. Whenever that changes (and the modal is hidden from view), the video URL is set to an empty string.

And this results in no video continuing to play when the video modal is dismissed.

To see how it works, watch this video with the sound on to hear how the audio ceases when the modal is dismissed and hidden from view.

Pretty useful, huh?

Conclusion

Small details can make for great user experiences, whether it's a video that stops playing once a user's scrolled past it or an animation that only happens when the elements are in view for a user.

While these little interactions once required a good deal of extra code and awareness of how long the main thread might be blocked, the introduction of the Intersection Observer API simplified things dramatically. Now, a single object allows a developer to specify an element to observe, a callback function to do something, and even fine tuned the function to only fire when certain conditions are met. It's quite useful in many scenarios, actually.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading. I hope this demo of how to use the Intersection Observer API helps you out in your future endeavors. I know I really appreciate great website experiences, and it's always good to have another tool in the arsenal to make them possible.

References & Further Resources

Change SVG Color with Help from CSS Filter

Mon, 05 Sep 2022 00:00:00 GMT

import YouTube from '../../src/components/YouTube.astro';

</figure>

Introduction

Working with talented designers is such a pleasure as a web developer. Not only do they come up with beautiful layouts for pages and simplify complex concepts into easy to understand visuals, but oftentimes they provide me the opportunity to learn a new web development technique to bring their visions to life.

I recently had such an opportunity while building a new About Us page for my company's marketing site. In the mockup, the page had a large quote by our founder, and the quotation marks around the text were stylized SVG images. When the designer gave me the SVGs, however, they came in a standard black color which didn't match the blue color in the design.

Before I asked my designer to recreate new SVGs to match the color in the design (or attempt to alter the SVG's color myself), I did a little digging online and found a useful CSS technique: filter. With filter, I was able to change the SVG's color myself - and change it to any hue I desired - no bothering my designer required.

In this post, we'll explore CSS filter and see how it can be used to alter SVG image colors to match colors in mockups.

I'll even share a handy website that will generate filter combinations to exactly replicate CSS hex colors.

Below is a video showing how the CSS filter affects the color of quotation mark SVGs - note how they change from black to light blue.

CSS Filter

Before I share the solution, let's take a closer look at CSS filter - it wasn't a technique that I was familiar with prior to this.

According to the MDN documents about CSS filter, it's used to apply graphical effects like blur or color shift to an element. Some standard CSS functions include:

blur - make the image fuzzier or sharper.
contrast - adjust the contrast of an image's colors.
grayscale - convert an image to grayscale.
drop-shadow - apply a blurred, offset version of the image, drawn in a specific color and composited below the image.

In addition to these standard functions, an SVG can also be referenced with a URL to an SVG filter element - essentially, a custom filter.

This article doesn't go into great detail about how the filter functions work because they're quite complex. If you want a full rundown of what they are and how they work, I recommend checking out the Mozilla documentation for more info.

Now what the documents fail to highlight is that multiple of these functions can be combined together in a single function to great effect - like, say, changing an SVG's color, which is what we're going to cover next.

Filters in action

<figcaption>The final quote with its light blue SVG quotation marks.</figcaption>

</figure>

Ok, let me show you how we can use these filter functions in your own code. The first thing you'll need is some HTML elements to apply this CSS to.

Set up the HTML, SVGs, and the img-quote class

The site I was working on happens to be made with Hugo, a popular, open source, static site generator written in Go. Although it's written in Go, it relies on templates to render the majority of the site's HTML, so the example code below should look relatively familiar to you if you know HTML.

A note on templates:

If you've ever used React's JSX components, Node.js with Pug or Handlebars, or Jekyll - Hugo's templates are similar: HTML elements with Go variables and functions sprinkled in with {{ }} to render the correct information wherever the template's injected.

Here's the code below for the quote section. For the purposes and clarity of this post, I've replaced the Go variables injected into the template with the generated HTML.

quote-template.html

    <section class="quote-section">
      <div class="container">
        <div class="row">
          <div class="col-12">
            <h2 class="text-center quote-text">
              <img
                class="quote-img"
                src="/images/about/quote-left-solid.svg"
                alt="decorative left quotation mark"
              />
              Complexity kills. It sucks the life out of developers.
              <div />
              It makes products difficult to plan, build, test, deliver, and support.
              <img
                class="quote-img"
                src="/images/about/quote-right-solid.svg"
                alt="decorative right quotation mark"
              />
            </h2>
            <div class="text-center">
              <h4 class="text-center quote-source">
                Ray Ozzie
                <span class="quote-source-role">
                  Founder & CEO, Blues Wireless
                </span>
              </h4>
            </div>
          </div>
        </div>
      </div>
    </section>

While this section of code is relatively small, it still deserves discussion. In addition to the HTML elements, there's quite a few CSS classes present, many of which are courtesy of Bootstrap, one of the original open source CSS frameworks for responsive web development.

Among the custom classes like quote-section and quote-img, which I used for custom styling, there are ones like container, row, col-12, and text-center. All of the latter examples are Bootstrap classes and serve to do things like have the quote text take up the entire width of the page: col-12, or center the text inside the <div>: text-center.

These classes are beside the point though, the one you want to focus on for this post is the quote-img class attached to the two <img> elements: these are the two quotation mark SVGs bookending the quote.

This quote-img class is how we're going to attach the CSS filter functions to both quotation mark SVGs and make sure they're both the correct color to match the design.

Add the CSS filter to quote-img

As I mentioned above, the SVGs my designer gave me were standard black, but in the design mockup, the quotes were a lighter shade of blue: hex color #94BBD0.

When I started looking for a solution to change the color of these SVGs, I came across this answer on StackOverflow.

While there were suggestions in the answer thread that amounted to modifying the SVG's source code directly (not something I was keen to do), but luckily I didn't have to.

The solution I landed on not only illustrated how CSS filter could be used to modify the SVG's color, it also linked to a helpful CodePen, which allowed me to input the hex color I wanted (#94BBD0), and it spit out the exact CSS filter combination to create it (plus an accuracy reading of just how close to perfect the CSS filter code was to matching the original hex color). And all I had to do was copy the code produced to use in my own CSS file. What an amazing timesaver!

<figcaption>CodePen to translate any SVG to a particular hex color: https://codepen.io/sosuke/pen/Pjoqqp. Bless the developer who went to the trouble of making this!</figcaption>

</figure>

So in my own SCSS file that was added to help style the About Us page's HTML, add the following class and the code copied from the CSS filter code generator.

NOTE: Even though this file shown in the example code is SCSS instead of pure CSS, for this post it shouldn't make a difference. It should be a direct 1:1 comparison.

about.scss

  .quote-img {
    height: 26px;
    filter: invert(84%) sepia(9%) saturate(1100%) hue-rotate(165deg)
      brightness(88%) contrast(83%);
    margin-bottom: 16px;
  }

A careful combination of multiple filter functions like invert, sepia and others, and voila - the exact shade of blue to match the hex code.

To make the image quotes the proper size and placement in relation to the text, in addition to changing their color, I also needed to set the height of the SVGs to 26px and give them a bottom margin of 16px to push them up to look like they were surrounding the text instead of sitting centered inline with the text.

And now the SVGs are correct color to match the mockup without designer intervention required - and can easily be changed in the future, if need be. How convenient is that?

Conclusion

Great designs have the power to capture people's attention and help them understand the benefits of a particular product or company - and many times they help the developers bringing these designs to life learn a new thing or two along the way.

As I was building a new page for my company's website that featured a large quote from our company founder, I was faced with just such a scenario where the stylized quotation marks were given to me as black SVG images, but they were shown in the design as being blue.

Before asking my designer to take the time recreate them in the appropriate color to match the design, I did some preliminary googling and found a much more flexible, code driven solution making great use of CSS filter.

And not only did the answer exist, but someone had also made a CodePen that could match any hex color! Just what I needed, and there for anyone to benefit from. The web development community can be so awesomely generous.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading. I hope this cool trick to convert SVGs from one color to another using CSS's filter comes in handy for you like it did for me. It's a great feeling as a developer to have the power to make these sorts of changes through code.

References & Further Resources

MDN Docs, CSS Filter
StackOverflow thread for CSS filter solution
CodePen site to match SVG to any hex color via CSS filter

Render Multiple Colored Lines on a React Map with Polylines

Mon, 08 Aug 2022 00:00:00 GMT

</figure>

Introduction

Last summer I started working for an Internet of Things startup, Blues Wireless, whose goal is to make IoT development easier by providing prepaid cellular Internet connectivity to any IoT device through the use of a Notecard, which transmits sensor data as JSON to a secure cloud, Notehub.

In a previous post, I showed how I used Next.js and React Leaflet to build an asset tracker map to display where a moving Notecard was (inside my car) in near real time. This exercise ended up coming in more handy than I expected when my parents' car was stolen out of their driveway over the Thanksgiving holiday and I'd stuck a Notecard in the backseat while visiting.

Although the Notecard was discovered shortly after the car was stolen and thrown out a window, for a short period of time we (and the police) were able to follow the car around town courtesy of my dashboard map, and this experience inspired me during a company-wide hackathon a few months later.

One thing that would have been very helpful during the period when the car was stolen was if the lines between points on the map could have been in some color besides the standard blue for easier identification of where the car was after it was taken. So for the hackathon I created a new dashboard with an "SOS mode" to not only render lines in red on the map after the SOS mode was enabled, but also increase the frequency of the Notecard taking location readings for better accuracy.

Today, I'll show you how to create a map with React Leaflet in Next.js with the ability to render different colored lines at the touch of button for asset tracking under normal or emergency conditions.

Set up a map component in Next.js app

NOTE: This article will not go through setting up a brand new Next project or an in-depth explanation of fetching asset tracker data from a Blues Wireless Notecard, as I've covered this already in this post.

To see the finished code, you can view my GitHub repo here

Install map project dependencies

The first thing we'll do in this tutorial is add a map to a Next project. This is going to require a few new npm packages added to our project: leaflet, react-leaflet and leaflet-defaulticon-compatibility.

Run the following lines in a terminal.

$ npm install leaflet react-leaflet leaflet-defaulticon-compatibility

NOTE: You'll also need react and react-dom as peer dependencies if they're not already in your project as well.

TypeScript Note:

If you're using TypeScript in your project, you'll also want to want to install the follow dev dependency to avoid TypeScript errors:

$ npm install @types/leaflet --save-dev

After installing our new project dependencies we'll set up the component to use them.

Generate a Mapbox token for the map's display style and add it to the project

For the map display that the asset tracker will be on, I chose to use Mapbox styles. It has a variety of styles to choose from, and developers can create their own Mapbox API tokens to access these styles by signing up for a free Mapbox account.

After you've signed up and created a new API token, copy the token value. In the Next.js app's next.config.js file at the root of the project, add the API token like so:

next.config.js

/** @type {import('next').NextConfig} */
module.exports = {
  reactStrictMode: true,
  env: {
    MAPBOX_ACCESS_TOKEN:
      "[MAPBOX_TOKEN]",
  },
};

From this file, Next can access the token when it needs to call the Mapbox API endpoint. Now we can get on with creating the <Map /> component in our project.

Create the `<Map>` component

<figcaption>This is how the map looks displaying asset locations during normal circumstances.</figcaption>

</figure>

As this is a React project, individual, reusable components are the name of the game, so create a new file named Map.tsx and paste in the following code.

The actual code is available by clicking the file title below.

Map.tsx

import {
  MapContainer,
  TileLayer,
  Marker,
  Popup,
  CircleMarker,
  Polyline,
} from "react-leaflet";
import "leaflet/dist/leaflet.css";
import "leaflet-defaulticon-compatibility/dist/leaflet-defaulticon-compatibility.css";
import "leaflet-defaulticon-compatibility";

const Map = ({
  coords,
  lastPosition,
  markers,
  latestTimestamp,
  sosCoords,
}: {
  coords: number[][];
  lastPosition: [number, number];
  markers: [number, number][];
  latestTimestamp: string;
  sosCoords?: number[][];
}) => {
  const geoJsonObj: any = coords;
  const sosGeoJsonObj: any = sosCoords;

  const mapMarkers = markers.map((latLng, i) => (
    <CircleMarker key={i} center={latLng} fillColor="navy" />
  ));

  return (
    <>
      <h2>Asset Tracker Map</h2>
      <MapContainer
        center={lastPosition}
        zoom={14}
        style={{ height: "100%", width: "100%" }}
      >
        <TileLayer
          url={`https://api.mapbox.com/styles/v1/mapbox/streets-v11/tiles/256/{z}/{x}/{y}@2x?access_token=${process.env.MAPBOX_ACCESS_TOKEN}`}
        />
        <Marker position={lastPosition} draggable={true}>
          <Popup>
            Last recorded position:
            <br />
            {lastPosition[0].toFixed(6)}&#176;,&nbsp;
            {lastPosition[1].toFixed(6)}&#176;
            <br />
            {latestTimestamp}
          </Popup>
          <Polyline pathOptions={{ color: "blue" }} positions={geoJsonObj} />
          <Polyline pathOptions={{ color: "red" }} positions={sosGeoJsonObj} />
          {mapMarkers}
        </Marker>
      </MapContainer>
    </>
  );
};

export default Map;

Let's briefly discuss what's happening here.

At the beginning of the file we import all the necessary React Leaflet components, the Leaflet CSS, and the Leaflet Default Icon Compatibility CSS and JS (this is recommended to get Leaflet's icons to work as expected).

Then we see the props the Map component expects:

coords - a list of arrays that have GPS latitude and longitude: this draws the connecting lines between map markers.
lastPosition - the most recent GPS latitude and longitude to display in the popup when the user clicks the icon on the map.
markers - another list of arrays that have GPS latitude and longitude to display the blue circles of previous places on the map where the tracker was in the past.
latestTimestamp - the most recent timestamp of GPS coordinates received (also for displaying in the popup on the map).
sosCoords - a separate list of GPS coordinates created when the "SOS mode" is enabled within the application: it will draw connecting lines between map markers in red.

Now turn your attention down to the JSX further down in the file.

The <MapContainer /> component is responsible for creating the Leaflet Map instance. Without this component, the map won't work, and we also define the map's center coordinates, its default zoom level, and basic styling for the component.

The <TileLayer /> component is where our Mapbox style and new API token come in. Choose whatever style suits you, replace the streets-v11 portion of the string, and make sure the Mapbox token is present in the next.config.js file, which I showed in the previous step. Without this component no map background for the coordinates will render - instead it will just be a blank canvas.

<Marker /> takes in the lastPosition prop to display the icon on the map of the tracker's last recorded position, and it wraps the <Popup /> component, the <Polyline /> components, and the list of <CircleMarker /> components.

The <Popup /> component is a nice looking tooltip that can display info. My <Popup /> shows the tracker's last GPS coordinates and time it was reported when a user clicks it.

The <Polyline /> components are where the coords list or sosCoords list of GPS coordinates are passed to draw the connecting lines between map markers. The Polyline object takes in positions, which in this case is either geoJsonObj or sosGeoJsonObj, and the pathOptions determines the color of the line rendered.

NOTE: At first, I tried to use the GeoJSON object to render the connecting lines, but there's no way to change line color midway through (as when SOS mode is enabled and lines need to go from blue to red), so multiple separate Polyline objects was the best way to achieve this objective.

And last but not least, the <CircleMarker >/ components, which are displayed in this component's JSX as {mapMarkers}.

NOTE: In order to get all the markers in the list to render as individual circles on the map, I had to create this little function to iterate over the list and generate all the circles, then inject that directly into the JSX.

Trying to iterate over all the values inside the JSX wouldn't work.

Now our Map component's been dissected, let's move on to populating the map with data and going from blue lines to red and back again.

Render the map in the Next.js app

The next step to getting this map working in our Next.js app is to import the Map component with the option of ssr: false.

The react-leaflet library only works on the client side so Next's dynamic import() support with no SSR feature must be employed to ensure the component doesn't try to render server-side.

Below is the code for the index.tsx file that this component will be displayed within, condensed for clarity. If you'd like to see the full code in GitHub, click the file name.

pages/index.tsx

// imports
import dynamic from "next/dynamic";
// other imports

type dataProps = {
// condensed for code brevity
};

export default function Home({ data }: { data: dataProps[] }) {
  // needed to make the Leaflet map render correctly
  const MapWithNoSSR = dynamic(() => import("../src/components/Map"), {
    ssr: false,
  });

 // logic to enable/disable sos mode and transform data into items needed to pass to map

  return (
    <div>
      {/* extra tracker app code */}
      <main>
        <h1>Notelink Tracker Dashboard</h1>
        {/* other tracker components */}
          <MapWithNoSSR
            coords={latLngMarkerPositions}
            lastPosition={lastPosition}
            markers={latLngMarkerPositions}
            latestTimestamp={latestTimestamp}
            sosCoords={sosCoords}
          />
        </div>
      </main>
    </div>
  );
}

// code to fetch tracker data: getStaticProps

Don't worry too much about the props being passed to the component yet - we'll set those up shortly.

Now our <Map /> component is being dynamically imported with Next's server-side rendering disabled, and the component can be used just like any other in the app.

Fetch data for the map

In my previous asset tracking dashboard article I went into great detail about how to create your own asset tracker to generate real data for the app using Blues Wireless hardware and fetching that data to the app via the Notehub cloud's API.

If you'd like to follow along there to build your own tracker and route data to Notehub, please be my guest.

For this post, I'll jump ahead to the part where we're already pulling data into the app via a Next.js getStaticProps API call. The JSON data from the Notehub cloud looks like this when it first arrives:

[
  {
    "uid": "d7cf7475-45ff-4d8c-b02a-64de9f15f538",
    "device_uid": "dev:864475ABCDEF",
    "file": "_track.qo",
    "captured": "2021-11-05T16:10:52Z",
    "received": "2021-11-05T16:11:29Z",
    "body": {
      "hdop": 3,
      "seconds": 90,
      "motion": 76,
      "temperature": 20.1875,
      "time": 1636123230,
      "voltage": 4.2578125
    },
    "gps_location": {
      "when": "2021-11-05T16:10:53Z",
      "name": "Sandy Springs, GA",
      "country": "US",
      "timezone": "America/New_York",
      "latitude": 33.913747500000014,
      "longitude": -84.35008984375
    }
  },
  {
    "uid": "3b1ef772-44da-455a-a846-446a85a70050",
    "device_uid": "dev:864475ABCDEF",
    "file": "_track.qo",
    "captured": "2021-11-05T22:22:18Z",
    "received": "2021-11-05T22:23:12Z",
    "body": {
      "hdop": 2,
      "motion": 203,
      "seconds": 174,
      "temperature": 22,
      "time": 1636150938,
      "voltage": 4.2265625
    },
    "gps_location": {
      "when": "2021-11-05T22:22:19Z",
      "name": "Doraville, GA",
      "country": "US",
      "timezone": "America/New_York",
      "latitude": 33.901052500000006,
      "longitude": -84.27090234375
    }
  },
  {
    "uid": "e94b0c68-b1d0-49cb-8361-d622d2d0081e",
    "device_uid": "dev:864475ABCDEF",
    "file": "_track.qo",
    "captured": "2021-11-05T22:40:04Z",
    "received": "2021-11-05T22:46:30Z",
    "body": {
      "hdop": 1,
      "motion": 50,
      "seconds": 41,
      "temperature": 21.875,
      "time": 1636152004,
      "voltage": 4.1875
    },
    "gps_location": {
      "when": "2021-11-05T22:40:05Z",
      "name": "Peachtree Corners, GA",
      "country": "US",
      "timezone": "America/New_York",
      "latitude": 33.9828325,
      "longitude": -84.21591015624999
    }
  },
  {
    "uid": "1344517c-adcb-4133-af6a-b1132ffc86ea",
    "device_uid": "dev:864475ABCDEF",
    "file": "_track.qo",
    "captured": "2021-11-06T03:04:07Z",
    "received": "2021-11-06T03:10:51Z",
    "body": {
      "hdop": 1,
      "motion": 126,
      "seconds": 218,
      "temperature": 12.5625,
      "time": 1636167847,
      "voltage": 4.1875
    },
    "gps_location": {
      "when": "2021-11-06T03:04:08Z",
      "name": "Norcross, GA",
      "country": "US",
      "timezone": "America/New_York",
      "latitude": 33.937182500000006,
      "longitude": -84.25278515625
    }
  }
]

Each JSON object in this array is a separate _track.qo motion event that displays the Notecard's current location and sensor readings. The part of the object that we care about in this particular post is the gps_location values: latitude, longitude, and the captured value. This is the data we'll need for the map.

Shortly we'll work on transforming this data to fit our <Map /> component's props - we'll handle that right after we create SOS mode for the app.

Configure SOS mode in the app

<figcaption>The SOS button to toggle SOS mode in the app.</figcaption>

</figure>

Before we transform this JSON data, we need to give our application the option to turn SOS mode on or off (which changes the color of the polylines rendered in the map).

To build this, we'll need a new state variable, function, and button in our index.tsx file.

pages/index.tsx

// imports
import { useState } from "react";
// more imports

type dataProps = {
// condensed for code brevity
};

export default function Home({ data }: { data: dataProps[] }) {
  // map component imported dynamically here

  const [isSosModeEnabled, setIsSosModeEnabled] = useState<boolean>(false);

  const toggleSosMode = () => {
    const newSosState = !isSosModeEnabled;
    if (newSosState === true) {
      localStorage.setItem("sos-timestamp", new Date());
      setIsSosModeEnabled(newSosState);
    } else {
      localStorage.removeItem("sos-timestamp");
      setIsSosModeEnabled(newSosState);
    }
  };

 // logic to transform data into items needed to pass to map

  return (
    <div>
      {/* extra tracker app code */}
      <main>
        <h1>Notelink Tracker Dashboard</h1>
        <button onClick={toggleSosMode}>
          SOS Mode
        </button>
        {isSosModeEnabled ? <p>SOS Mode Currently On</p> : null}
        {/* other tracker components */}
          <MapWithNoSSR
            coords={latLngMarkerPositions}
            lastPosition={lastPosition}
            markers={latLngMarkerPositions}
            latestTimestamp={latestTimestamp}
            sosCoords={sosCoords}
          />
        </div>
      </main>
    </div>
  );
}

// code to fetch tracker data: getStaticProps

In the file above, we're adding a new isSosModeEnabled boolean: this will let the app know whether new location events are happening during an emergency tracking situation or not.

Next, we create a new function called toggleSosMode(). This function will change the state of the isSosModeEnabled and also store a timestamp named sos-timestamp in the browser's local storage. I'm storing this timestamp in local storage so it can be compared to events that reach the app after SOS mode has been enabled and the app will know whether it needs to render the polylines on the map in red or blue. We'll get to the logic for this part in the following section.

Lastly, in the JSX for the component, we'll make a new <button> element and attach the toggleSosMode() function to its onClick() method. I also added a <p> tag beneath the button to display when SOS Mode is in effect in the app.

Pressing the button after app mode will turn it on, pressing it again will turn it off.

NOTE: In the actual repo there's also logic to update the Notecard via Notehub to take more frequent location readings, and handle if SOS mode was already enabled and the browser window was closed or app connection was lost, but for clarity in this tutorial, I've eliminated both of those considerations.

Now that we can turn SOS mode on and off in the browser at will, it's time to take our location data and transform it to render in our map.

Reshape the tracker event data

Our index.tsx file is going to need some more state variables to fulfill all the different data props the <Map /> component needs. Once again, I've condensed the logic to make this file easier to read, but you can always click the file name to see its full contents online.

pages/index.tsx

// imports
import { useEffect, useState } from "react";
import dayjs from "dayjs"; // for ease of date formatting 
// more imports

type dataProps = {
// condensed for code brevity
};

export default function Home({ data }: { data: dataProps[] }) {
  // map component imported dynamically here

   const [lastPosition, setLastPosition] = useState<[number, number]>([
    33, -84,
  ]);
  const [latestTimestamp, setLatestTimestamp] = useState<string>("");
  const [latLngMarkerPositions, setLatLngMarkerPositions] = useState<
    [number, number][]
  >([]);

  // isSosEnabled boolean here
  const [sosCoords, setSosCoords] = useState<number[][]>([]);

  /* runs as soon as the location data is fetched from Notehub API 
    or when the sos mode is toggled on or off with the button */
  useEffect(() => {
    const latLngArray: [number, number][] = [];
    const sosLatLngArray: [number, number][] = [];
    if (data && data.length > 0) {
      data
        .sort((a, b) => {
          return Number(a.captured) - Number(b.captured);
        })
        .map((event) => {
          let latLngCoords: [number, number] = [];
          let sosLatLngCoords: [number, number] = [];
          if (!isSosModeEnabled) {
            latLngCoords = [
            event.gps_location.latitude,
            event.gps_location.longitude,
            ];
            latLngArray.push(latLngCoords);
          } else {
            const localSosTimestamp = localStorage.getItem("sos-timestamp");
            if (Date.parse(event.captured) >= Date.parse(localSosTimestamp)) {
                sosLatLngCoords = [
                  event.gps_location.latitude,
                  event.gps_location.longitude,
                ];
              sosLatLngArray.push(sosLatLngCoords);
            } else {
              latLngCoords = [
                event.gps_location.latitude,
                event.gps_location.longitude,
              ];
              latLngArray.push(latLngCoords);
            }
          }
        });
      const lastEvent = data.at(-1);
      let lastCoords: [number, number] = [0, 1];
      lastCoords = [
        lastEvent.gps_location.latitude,
        lastEvent.gps_location.longitude,
      ];
      setLastPosition(lastCoords);
      const timestamp = dayjs(lastEvent?.captured).format("MMM D, YYYY h:mm A");
      setLatestTimestamp(timestamp);
    }
    if (sosLatLngArray.length > 0) {
      setSosCoords(sosLatLngArray);
    }
    setLatLngMarkerPositions(latLngArray);
  }, [data, isSosModeEnabled]);

  // toggleSosMode function  

  return (
    <div>
      {/* extra tracker app code */}
      <main>
        <h1>Notelink Tracker Dashboard</h1>
        {/* other tracker components */}
          <MapWithNoSSR
            coords={latLngMarkerPositions}
            lastPosition={lastPosition}
            markers={latLngMarkerPositions}
            latestTimestamp={latestTimestamp}
            sosCoords={sosCoords}
          />
        </div>
      </main>
    </div>
  );
}

// code to fetch tracker data: getStaticProps

In our main component, once the data is fetched from Notehub, we set the following new React useState variables to hold the data to pass to the <Map /> component.

lastPosition, latestTimestamp, latLngMarkerPositions, and sosCoords, are the new state variables we'll need at our disposal.

After those states are declared, the useEffect() function will run whenever data is fetched from Notehub (on component mount) or when the app's SOS mode is toggled. Inside the function, the events from Notehub are sorted and iterated through.

If the isSosModeEnabled boolean is true, the sos-timestamp is fetched out of the browser's local storage, and the date of that timestamp is compared to the captured timestamp from each event so the event can be properly sorted into either the sosLatLngArray list or the latLngArray list.

Once those local arrays are assembled inside of the useEffect(), they're set equal to the state variables latLngMarkerPositions and sosCoords.

If isSosModeEnabled is false, then all the events are added to the latLngArray list automatically.

The other variables lastPosition and latestTimestamp are set simply by pulling the last event off of the sorted data array and extracting the properties from it.

Then all these variables are passed to the <Map /> component, and it knows what do to from there regarding markers, popups, and line colors.

Test it out

Ok! I think we're reading to test out our map and multicolored lines!

If you're using our mocked data instead of real-time data streaming in from the Notehub cloud, the easiest way to test the app is to toggle SOS mode on via the button in the browser, then adjust the timestamp in the browser's DevTool local storage to be before at least some of the events captured timestamps.

If everything goes according to plan, you should end up seeing a combination of colored lines depending on when the SOS mode's time is set to and when the event occurred.

<figcaption>When SOS mode is on, new events that occurred after it was enabled show up as red lines.</figcaption>

</figure>

Please keep in mind, this app was built in a day so it's rough around the edges and certainly not ready for prime time. I'd recommend forking it and giving it some extra polish and testing before putting it into prod, if you're so inclined.

And there you have it: multicolored lines in a map in a React application. Not too shabby for a day's work.

Conclusion

After I joined an IoT startup last summer, I started building web apps to reflect the data being captured and transferred by our IoT devices in the real world and sent to the cloud, and during a company-wide hackathon I built a dashboard that not only displayed location data but also had the ability to change location lines from blue to red at will.

It seemed like a nice feature to improve readability of the map in some sort of emergency situation.

Next.js combined with the React Leaflet library made it all possible, and within the timeframe I had a working (if very rough) prototype to show my coworkers. It was a lot of fun to work on, and I learned a bunch of new things along the way. That's my idea of a successful hackathon.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading. I hope you found it useful to see how to set up an interactive map in Next.js and render multicolored travel lines between different location points depending on the situation. Happy mapping!

References & Further Resources

Hackster.io original asset tracker project
SOS Asset Tracker GitHub repo
Leaflet documentation
React Leaflet documentation
Mapbox site
Blues Wireless website

Use Secret Environment Variables in GitHub Actions

Mon, 04 Jul 2022 00:00:00 GMT

</figure>

Introduction

Earlier this year, my development team and I were in the midst of building a new open source application to help our users get up and running faster with the hardware the Internet of Things (IoT) company I work for, sells.

I work for an IoT startup called Blues Wireless, whose mission is to make IoT development simpler and easier through the use of Notecards - prepaid cellular devices that can be embedded into any IoT project to transmit sensor data to a secure cloud: Notehub - that can then forward that data on to any public cloud or application.

The idea is that once our users set up their new IoT devices, they can fork our starter repo, which is already designed to accept and display certain sensor data in charts and graphs, and then modify the code as they customize their devices further with new sensors or data visualizations.

Keeping ease of setup and deployment in mind, we built the app using the self-contained Next.js framework, hosted a demo example of the app on Netlify, and used GitHub Actions to control the automated CI/CD workflows (build, test, deploy, etc.).

One tricky thing I encountered when working with GitHub Actions was how to add sensitive environment variables that were used locally to run the app and its integration and end-to-end tests, while still keeping them secret (and out of GitHub's source control).

In this article, I'll teach you how to add "secret" environment variables to a GitHub Actions workflow so steps requiring dynamic, sensitive variables (like automated test suites) can successfully run in the CI/CD pipeline just as they do in a local development environment.

What is GitHub Actions?

Let's step back for just a minute to talk about GitHub Actions in case you haven't had a need to work with them yet.

Back in 2018, GitHub released a new feature called GitHub Actions, designed to give developers the ability to automate their workflows across issues, pull requests, and other events.

In 2019 it introduced full continuous integration / continuous deployment (CI/CD) to GitHub Actions as well.

These tools make it easier to automate how developers build, test, and deploy projects on any platform without the need for a devops expert on a development team. Among the advantage GH Actions has over other CI/CD tools is that if a repo is already in GitHub (which it probably is), it's very straightforward to set up workflows to handle all of these sorts of tasks (and more) in an automated fashion.

If you've ever used another CI/CD platform like Jenkins, CircleCI, TeamCity, or a handful of other tools, GitHub Actions should feel similar to you once you start working with it.

Here's a sample diagram of what a simple CI/CD pipeline might look like.

<figcaption>Continuous integration diagram courtesy of <a href="https://resources.github.com/ci-cd/">GitHub Resources: CI/CD</a></figcation>

</figure>

Going into all the specifics of how GH Actions works is beyond the scope of this blog, but hopefully I've given you a good enough primer to understand the material we'll cover now.

In the Repo, Set the Environment Variables for GH Actions

The place to begin setting environment variables for GH Actions to use is inside the repo itself.

Identify the Local Environment Variables to Replicate in GitHub

In our particular repo, we have four dynamic environment variables that the app (and its test suites) require to run: HUB_BASE_URL, HUB_AUTH_TOKEN, HUB_PROJECT_UID, and HUB_PRODUCT_UID,

All of these variables relate to unique project values that are discoverable in Notehub, the cloud application where IoT sensor data is delivered via the Notecard.

HUB_BASE_URL is the URL to reach the Notehub API to fetch data.
HUB_AUTH_TOKEN is an authorization token required to access data in Notehub.
HUB_PROJECT_UID is the project ID in Notehub where the sensor data is sent.
HUB_PRODUCT_UID is the sensor device's unique ID - as one project can have many devices sending data to it.

Although not all of these environment variables are "top secret", since there's just four of them, it made more sense to keep them all grouped together in the repo instead splitting them up into "public" and "private" variables located in separate places in the code base.

Since this is a Next.js application, the way we locally store these sensitive variables that should not be checked in to source control (i.e. GitHub) is through the use of .env files.

For the app, the following sample code controls the environment variables read by Next.js when it starts up locally.

.env.local

HUB_BASE_URL=https://api.notefile.net
HUB_AUTH_TOKEN=[YOUR_AUTH_TOKEN_HERE]
HUB_PROJECT_UID=[YOUR_NOTEHUB_PROJECT_UID_HERE]
HUB_PRODUCT_UID=[YOUR_NOTEHUB_PRODUCT_UID_HERE]

These variables are then accessed inside the application using the process.env syntax. For instance, to get the HUB_PROJECT_UID value inside of the app's running code, it's simply a matter of writing process.env.HUB_PROJECT_UID in the JavaScript file where it's required.

Ok, so we've identified our environment variables that are dynamic and also should not be checked into GitHub, let's let GitHub Actions know it needs to look for them too when it runs its CI workflows.

Add the Environment Variables to the GitHub Actions YAML File

In order for GitHub to know it has a GitHub Action workflow to run for a repo, the following folders should exist at the root of the repo: .github/workflows/.

A YAML file will be contained therein: ours is named CI-PRs-and-main.yml, but you can name it anything that makes sense to you.

Inside of the YAML file, there will be a variety of steps listed out and for the steps requiring these secrets, we'll add the needed environment variables like so:

NOTE: If you'd like to see the full GitHub Actions file in its repo, you can click the file title below.

CI-PRs-and-main.yml

# more GitHub Actions steps above...
- name: Run unit tests
  env:
    HUB_AUTH_TOKEN: ${{ secrets.HUB_AUTH_TOKEN }}
    HUB_BASE_URL: ${{ secrets.HUB_BASE_URL }}
    HUB_PROJECT_UID: ${{ secrets.HUB_PROJECT_UID }}
    HUB_PRODUCT_UID: ${{ secrets.HUB_PRODUCT_UID }}
  run: yarn test:coverage
- run: yarn lint
- name: Cypress run # our e2e tests
  uses: cypress-io/github-action@v2
  with:
    env: configFile=staging
    start: yarn start
  env:
    HUB_AUTH_TOKEN: ${{ secrets.HUB_AUTH_TOKEN }}
    HUB_BASE_URL: ${{ secrets.HUB_BASE_URL }}
    HUB_PROJECT_UID: ${{ secrets.HUB_PROJECT_UID }}
    HUB_PRODUCT_UID: ${{ secrets.HUB_PRODUCT_UID }}
# more GitHub Actions steps below...

If you notice there are two steps in this YAML file that require the sensitive environment variables: Run unit tests and Cypress run.

Declare the env property and each needed environment variable

To tell GH Actions these custom environment variables should exist, we declare the env property inside the steps that need the variables.

Then indent and list each secret variable needed (and how the repo reads them). For example, since our repo reads the secret variable HUB_AUTH_TOKEN locally, it makes sense to name the env property to be read during the GitHub Actions workflow by the same name.

  env:
    HUB_AUTH_TOKEN:

Set the ${{ secrets.MY_SECRET }} value

After listing out the environment variable names for GitHub Actions, use the syntax ${{ secrets.NAME_OF_SECRET_IN_GITHUB }} to assign a value stored in GitHub Actions by the same name.

I'll go into how to set this value remotely in the next section.

In the GitHub Repo Set Those Sensitive Environment Variables

Now we're ready to move on to setting the secret variables in GitHub.

Access the Repo's Settings Tab

Head to your repo in GitHub that needs the variables and click on the Settings tab.

Inside of Settings, go down the sidebar to the Secrets dropdown and click on it to access the Actions page. The page should look something like this image below.

</figure>

In my Actions Secrets page, there are already a few Repository Secrets present - this is what we'll be adding to to run in the GH Actions workflow. Click on the New repository secret button in the upper right corner of the screen.

Set New Environment Secrets

A new screen will appear to add new secrets to the repository. Here is where the environment variable names defined in the GitHub Actions YAML will be replicated and their values defined.

Using our example variables from above, we'd define a new variable named HUB_PROJECT_UID as the secret's name in the Name input, then add the actual project's value (as defined in Notehub, in this case) into the Value text area.

This corresponds to the ${{ secrets.HUB_PROJECT_UID }} value in the YAML file.

</figure>

Click the Add Secret button after double checking the secret name and value are correct. Once the secret's saved it's encrypted in GitHub and you'll be hard pressed to see what its original value was.

NOTE: If you do realize later a secret is incorrect or needs to be updated, not to worry, secret values can be updated by clicking the Update button next to the secret's name on the main Actions Secrets page.

You still won't be able to see what the old value of the secret was, but there is a path forward to make changes.

Great! The GitHub Actions file knows what environment variables to inject for certain steps of the workflow, those environment variables and their values have been defined in the repo's settings tab, it's time to test it out.

Run Your GitHub Actions Workflow and Watch it Pass

For our repo, the GitHub Actions workflows are designed to run whenever a user pushes code to a pull request or to the main branch, as defined here:

CI-PRs-and-main.yml

# more GitHub Actions code above
on:
  pull_request:
  push:
    branches:
      - main
# more GitHub Actions code below

If you have an already existing workflow that failed, simply go to the repo's Actions tab, click on the failing workflow, and click the Re-run all jobs button on the main workflow's screen.

If not, perform whatever action is needed to trigger a new workflow to run.

Either way, this is the end result you should see when the secret environment variables set are successfully read out.

</figure>

Notice in the image that among all the steps listed out during the course of the workflow, the two that required the sensitive environment variables, Run Unit Tests and Cypress run, both successfully passed.

No sensitive info or secrets revealed in the process.

Conclusion

With the introduction of GitHub Actions a few years ago, GitHub significantly reduced the complexity and expertise needed by development teams to take advantage of the benefits continuous integration / continuous deployment pipelines provide. One thing that's always been a problem though is how to handle environment variables that should be kept secret from the general code base for their sensitive nature - the very kinds of secrets required for apps (and automated test suites) to run.

GitHub figured out a way to handle that though: secrets set in the repo's settings and encrypted as soon as they're set. Instead of having to figure out how to keep secrets safe through the use of .gitignore JSON files, build pipeline vaults, or other, non-intuitive, difficult-to-manage systems, GitHub can simply store the variables and access them in any necessary workflow step using the ${{ secrets.MY_SECRET_HERE }} syntax. Much better!

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading. I hope this helps you out when building your own GitHub Actions workflows that require sensitive environment variables: CI/CD is the best way of building quality software today, and GitHub is making it easier and easier for everyone to achieve that.

References & Further Resources

Merge JavaScript Objects in an Array with Different Defined Properties

Mon, 06 Jun 2022 00:00:00 GMT

</figure>

Introduction

At work recently, I encountered an interesting problem: I needed to merge multiple JavaScript objects together in an array based on their matching keys.

It sounds relatively straightforward at first, but there's more to the story that prevented me using the spread operator or Object.assign() to combine these objects.

Each of these objects might share a common key - it wasn't guaranteed, and
Although they all had the same properties in each object, not all of these properties were defined in every object - the object could have property: value or it could be property: undefined.

I needed to combine all the properties of objects that shared a common key along with their defined values into a new single object. And to make it just a smidge more difficult I needed it to work in TypeScript. 😅

So today, I'll show you how to use JavaScript's reduce() function to merge two (or more) objects in an array with different defined properties.

Background: `motion.qo` and `air.qo` events

For those of you curious as to when this sort of thing comes up in real life, here's a little more context about my situation.

I work as a software engineer at an Internet of Things (IoT) startup named Blues Wireless. Our main mission is to make IoT development simpler, regardless of if there's a reliable Internet connection or not. Blues does this via Notecards - prepaid cellular devices that can be embedded into any IoT device "on the edge" to transmit that sensor data as JSON to a secure cloud: Notehub.

In addition to producing this hardware and providing the secure cloud connection to send IoT data to, we also build software applications our users can connect their devices to and see the sensor data in charts and graphs.

<figcaption>Here's an example of the charts displaying various sensor data.</figcaption>

</figure>

One of the web apps we were building involved sensors that reported two types of data:

air quality readings like temperature, humidity, sensor voltage, and pressure, and
motion readings like current count of motions detected and total motions detected over time.

So a single sensor is generating two different JSON payloads that are sent from a Notecard to Notehub, then Notehub routes them to our web app to get rendered in the charts.

Here's a sample of some of the events delivered to the app.

Example sensor events

const sensorEvents = [
  {
    sensorId: 'abc',
    id: '1',
    humidity: undefined,
    pressure: undefined,
    temperature: undefined,
    voltage: undefined,
    count: 3,
    total: 32,
    lastActivity: '2022-02-10T00:41:11Z'
  },
  {
    sensorId: 'abc',
    id: '2',
    humidity: 17.609375,
    pressure: 96.768734,
    temperature: 26.703125,
    voltage: 3.24,
    count: undefined,
    total: undefined,
    lastActivity: '2022-02-15T21:59:18Z'
  },
  {
    sensorId: 'def',
    id: '3',
    humidity: undefined,
    pressure: undefined,
    temperature: undefined,
    voltage: undefined,
    count: 866,
    total: 1776,
    lastActivity: '2022-02-15T22:00:03Z'
  }
]

As you can see, the first two objects in the array share the same sensorId of abc, but they have different defined properties.

The first object has motion properties (and undefined air quality properties), the second object has defined air quality properties (and undefined motion properties), and the third object has a different sensorId entirely; that is a motion reading from another sensor.

Which brought me to my problem: how to combine the objects with the same sensorId and get all the defined properties from each object.

So, let's get to it.

Merge like objects with a `mergeObject()` function and `reduce()`

With the amount of times I've said "merge" and "combine" in this post, you may already have figured out that the JavaScript array function reduce() might come into play here, and it does.

At its most basic, reduce() goes through each element in an array and adds it to an accumulator of all the values preceding it, until just one value is returned at the end. The same thing can be done with an array of objects, with just a little more logic.

If you'd like a more thorough explanation of reduce(), I encourage you to check out the Mozilla documentation.

Their examples are great, and I still visit it regularly for all sorts of topics.

Below is the TypeScript-flavored solution I came up with to take an array of objects, find the ones that share a common key, and return a new array with those objects combined.

interface HasSensorId {
  sensorId: string;
}

// merge objects with different defined properties into a single obj
const mergeObject = <V>(A: any, B: any): V => {
  let res: any = {};
  Object.keys({ ...A, ...B }).map((key) => {
    res[key] = B[key] || A[key];
  });
  return res as V;
};

// use Map functions `get`, `set`, and `values`
// take each event, and make it into a new Map() obj where sensorId is the key
// regardless of if there's already a value for that key, run the merge function
// set the key and the new merge var as the key, value for the map
const reducer = <V extends HasSensorId>(
  groups: Map<string, V>,
  event: V
) => {
  const key = event.sensorId;
  const previous = groups.get(key);
  const merged: V = mergeObject(previous || {}, event);
  groups.set(key, merged);
  return groups;
};

// run the sensor events through the reducer and then pull only their values into a new Map iterator obj
const reducedEventsIterator = sensorEvents
  .reduce(reducer, new Map())
  .values();

// transform the Map iterator obj back into an array
const reducedEvents = Array.from(reducedEventsIterator);

Ok, I realize there's quite a few functions above, so an explanation is in order.

HasSensorId

The HasSensorId interface is pretty direct: it means that wherever this interface is referenced, the object's must have a string of sensorId among the properties.

Now let's walk through the rest of these functions one by one.

mergeObject()

const mergeObject = <V>(A: any, B: any): V => {
  let res: any = {};
  Object.keys({ ...A, ...B }).map((key) => {
    res[key] = B[key] || A[key];
  });
  return res as V;
};

This mergeObject() function is the one that actually does the combining of objects.

The A and B objects that this function takes in are the two objects with a common sensorId property, and the res variable will be the newly returned object of their properties (and values) combined.

The first thing that happens is the two objects are combined using the spread operator, and then just their keys are extracted into its own array using Object.keys(). If you passed the sample sensorEvents array into this function, the Object.keys() array would look like this:

[
  'sensorId',
  'humidity',
  'pressure',
  'temperature',
  'voltage',
  'count',
  'total',
  'lastActivity'
]

After the keys are extracted, each key is added to the new res object as a key, and if the B object's value at that key exists (B[key]), it's added as the value for the res object, otherwise, the A object's value is added instead (even if it's undefined).

The final res object returned looks like this:

{
  sensorId: 'abc',
  humidity: 17.609375,
  pressure: 96.768734,
  temperature: 26.703125,
  voltage: 3.24,
  count: 3,
  total: 32,
  lastActivity: '2022-02-15T21:59:18Z'
}

And finally the V and any references are part of the TypeScript implementation. From looking at the rest of the code, we can see V is an extension of the HasSensorId interface, which means any event object will have sensorId as a property. And any means the objects passed in to the function could have any properties (or not), so it's flexible.

Ok, on to the next function!

reducer()

const reducer = <V extends HasSensorId>(
  groups: Map<string, V>,
  event: V
) => {
  const key = event.sensorId;
  const previous = groups.get(key);
  const merged: V = mergeObject(previous || {}, event);
  groups.set(key, merged);
  return groups;
};

The reducer() function makes excellent use of true Map objects in JavaScript. It uses the HasSensorId interface as well, ensuring that every object will have a sensorId property.

Each event object passed into the function, has its sensorId turned into a key variable. The previous variable relies on Map's get() function to fetch any other Map objects that have that same sensorId key, and if another object is found, the new event is combined with it when the mergeObject() function is called. If there's no matching previous object, an empty object is passed to mergeObject(), and a new merged variable is still returned.

Finally, the Map set() function adds the key and merged variable to the groups Map object.

If our three sensorEvents are run through this function, the final groups object that emerges looks like this:

Map(2) {
  'abc' => {
    sensorId: 'abc',
    humidity: 17.609375,
    pressure: 96.768734,
    temperature: 26.703125,
    voltage: 3.24,
    count: 3,
    total: 32,
    lastActivity: '2022-02-15T21:59:18Z'
  },
  'def' => {
    sensorId: 'def',
    humidity: undefined,
    pressure: undefined,
    temperature: undefined,
    voltage: undefined,
    count: 866,
    total: 1776,
    lastActivity: '2022-02-15T22:00:03Z'
  }
}

And so we move on to the penultimate part of this code example, reducedEventsIterator.

reducedEventsIterator

const reducedEventsIterator = sensorEvents
  .reduce(reducer, new Map())
  .values();

The reducedEventsIterator variable is where we call reduce() on the sensorEvents array. The reducer() function gets passed as the first argument each event in the array must run through, and a new Map() object is passed as the accumulator object where they will all be gathered together at the end.

Before the Object.values() method is chained on to the end, the array looks like this:

[
  [
    'abc',
    {
      sensorId: 'abc',
      humidity: 17.609375,
      pressure: 96.768734,
      temperature: 26.703125,
      voltage: 3.24,
      count: 3,
      total: 32,
      lastActivity: '2022-02-15T21:59:18Z'
    }
  ],
  [
    'def',
    {
      sensorId: 'def',
      humidity: undefined,
      pressure: undefined,
      temperature: undefined,
      voltage: undefined,
      count: 866,
      total: 1776,
      lastActivity: '2022-02-15T22:00:03Z'
    }
  ]
]

After .values() is called, the final reducedEventsIterator looks more like this:

[Map Iterator] {
  {
    sensorId: 'abc',
    humidity: 17.609375,
    pressure: 96.768734,
    temperature: 26.703125,
    voltage: 3.24,
    count: 3,
    total: 32,
    lastActivity: '2022-02-15T21:59:18Z'
  },
  {
    sensorId: 'def',
    humidity: undefined,
    pressure: undefined,
    temperature: undefined,
    voltage: undefined,
    count: 866,
    total: 1776,
    lastActivity: '2022-02-15T22:00:03Z'
  }
}

Which brings us to the final step: transforming this Map of objects back to a proper JavaScript array of objects (and all the benefits that entails - arrays are much easier to manipulate).

The final merged array: reducedEvents

const reducedEvents = Array.from(reducedEventsIterator);

Arguably the simplest part of this whole tutorial, the Map of objects is transformed back into an array simply by passing it into the Array.from() method.

Beware Array.from() for copying deep objects

If you have deeply nested objects in some sort of array-like structure that you need to copy, Array.from() probably won't work for you. It only makes shallow-copied instances of arrays.

So if we pass the example sensor events displayed above into our reducedEventsIterator() function here, this is what the final result looks like:

[
  {
    sensorId: 'abc',
    humidity: 17.609375,
    pressure: 96.768734,
    temperature: 26.703125,
    voltage: 3.24,
    count: 3,
    total: 32,
    lastActivity: '2022-02-15T21:59:18Z'
  },
  {
    sensorId: 'def',
    humidity: undefined,
    pressure: undefined,
    temperature: undefined,
    voltage: undefined,
    count: 866,
    total: 1776,
    lastActivity: '2022-02-15T22:00:03Z'
  }
]

And there you have it: an array of freshly merged objects with all available properties present.

Conclusion

One reason I enjoy being a software engineer is because of all the interesting problems I get to solve regularly. Lately, when I was building a web app to display data from IoT sensors at work, I ran into a more unique problem: taking an array of sensor events and merging the events together if they belonged to the same sensor. The catch was, all the events had the same properties, but depending on which event you were looking at, half the properties had values of undefined.

So instead of being able to use the spread operator or Object.assign() to get these events merged into one object, I had to get more creative with a few extra functions and the reduce() method. But it worked - even with TypeScript in the mix.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading. I hope this little bit of JavaScript helps you out in future - who'd have thought merging objects could be so interesting, right?

References & Further Resources

Spread operator Mozilla docs
Object.assign() Mozilla docs
Map object Mozilla docs
Blues Wireless website
JavaScript array method reduce()

Use CSS Grid to Make a Fixed Sidebar with Scrollable Main Body

Sat, 14 May 2022 00:00:00 GMT

import YouTube from '../../src/components/YouTube.astro';

</figure>

Introduction

Last year I authored a web development course on modernizing enterprise React applications. It was a big undertaking, but I'm proud to have put out a comprehensive guide to show developers just what it takes to build and maintain a large scale, long-running React application: tooling, testing, refactoring, design systems, and more.

For this course, I built a React project similar to what you might find when joining an existing enterprise company: it's an internal application used by product managers to manage assortments of tools and other home improvement items and select which ones should be shipped to stores to be sold. I named it "Hardware Handler".

The project started out using an outdated version of React, having no tooling for easier setup, no automated tests, no React Hooks, no state management, no component library, and so on and so forth. As the course progressed through each module, I focused one aspect of the project and showed how to update the project to today's high quality software standards: adding React Testing Library and Jest for unit tests or refactoring class-based React components to use Hooks, for example.

One interesting thing I encountered while building the sample project is that I wanted a product page that showed all the possible products available, but also have a fixed sidebar with filtering options, so users could filter down products by a particular department or a particular brand. Now you might expect that I would reach for some CSS like position: fixed or position: absolute to keep this sidebar in place and essentially remove the sidebar from the normal flow of the DOM, but I didn't need to. Instead, all I needed were a few simple CSS Grid properties.

If you're not familiar with CSS Grid, it's a relatively new addition to CSS, and it excels at dividing a page into major regions and enabling an author to align elements into columns and rows.

I'd highly recommend checking out the MDN docs and the CSS-Tricks "A Complete Guide to Grid", for more CSS Grid info.

Today, I'll show you how to build your own fixed sidebar in an application while still allowing the main body content to scroll freely with the help of CSS Grid.

Below is a video of what the final page looks like: notice the page title and filters on the lefthand side stay in place while the product cards scroll down in the main content space.

Set up the `ProductList` component's HTML (or JSX)

Before we can start applying the CSS Grid Layout to our page, we must actually code a page full of HTML elements to apply the grid to.

My project was built with the React framework, so the example code I'll be walking through would normally contain React Hooks, state, functions, and JSX, but for this blog post, everything besides the JSX is largely unnecessary, so I've cut it out.

It should be relatively straightforward to translate the remaining JSX into HTML or another JS-powered framework, if you so desire.

Here's a condensed version of the actual ProductList component, please note I've eliminated much of the React-specific code and things like error handling messages, loading components, etc. to focus on the HTML that our CSS will apply to.

ProductList.js

// other imports for React, components, constants, etc.
import './ProductList.css';

const ProductList = () => {
  // all sorts of state variables here: products, loading, errors, etc.

  // logic to facilitate adding items to checkout

  // logic for filtering products based on brand name or department 

  return (
    <div className="product-list-container">
      <section className="filter-wrapper">
        <p className="filter-title">Filter by Department</p>
        <div className="filter-data">
          {filtersByDepartment.map((filter) => (
            <span key={filter.id} className="filter-item">
              <label htmlFor={filter.id}>{filter.name}</label>
              <input
                className="filter-checkbox"
                id={filter.id}
                type="checkbox"
                checked={activeFilter.includes(filter.id)}
                onChange={() => onFilterChange(filter.id)}
              />
            </span>
          ))}
        </div>
        <p className="filter-title">Filter by Brand</p>
        <div className="filter-data">
          {filtersByBrand.map((filter) => (
            <span key={filter.value} className="filter-item">
              <label htmlFor={filter.value}>{filter.name}</label>
              <input
                className="filter-checkbox"
                id={filter.value}
                type="checkbox"
                checked={activeFilter.includes(filter.value)}
                onChange={() => onFilterChange(filter.value)}
              />
            </span>
          ))}
        </div>
      </section>
      <h1 className="product-list-header">My Products</h1>
      <section className="products-container">
        <div className="product-list-product-wrapper">
          {filteredList.map((product) => (
            <Product
              key={product.id}
              product={product}
              addItemToCheckout={addItemToCheckout}
            />
          ))}
        </div>
      </section>
    </div>
  );
};

export default ProductList;

I cut out a lot of unneeded noise, but that's still plenty happening in this component to render both a list of all the products and two different filtering options, so let's talk through what's happening in this code snippet.

The first section tag is wrapping our various filters:

One filtering option users have is to filter products by department, hence the array-based state variable titled filtersByDepartment,
And the other filtering option is to filter products by brand name: filtersByBrand.

Each of these arrays of data produces a set of checkboxes that users can check to narrow down the list of products being displayed on screen at any one time.

Which brings us to the final section of the JSX where the filteredList variable is being rendered - that variable is a list of all the products that fit into the selected filter criteria.

Also take note of the classes attached to many of these elements: product-list-container, filter-wrapper, product-container - they'll come into play in the next section when we write the CSS.

I've omitted the filtering logic that's also in this component because it isn't central to illustrating the point in this post.

And that's what's going on in this component.

Add CSS Grid to Our HTML

With our JSX (or HTML) set up, we can get to the CSS portion of this blog post. This is the entirety of the CSS contained in our ProductList.css file - there are some things that are being inherited from other global CSS - things like background-color or font-size, but by and large, this is all you'll need to achieve the desired effect.

ProductList.css

/* styling for the whole product list container, including the filters on the side */
.product-list-container {
  display: grid;
  grid-template-rows: 55px calc(100vh - 55px);
  grid-template-columns: 220px auto;
  grid-template-areas:
    'filter header'
    'filter products';
}

/* styling for just the main list of products displayed on screen */
.product-list-header {
  display: flex;
  align-items: center;
  justify-content: center;
  grid-area: header;
}

.products-container {
  grid-area: products;
  overflow: auto;
}

.product-list-product-wrapper {
  display: flex;
  flex-wrap: wrap;
  margin: auto;
}

/* styling for the filters */
.filter-wrapper {
  padding: 10px;
  background-color: #ffcfae;
  grid-area: filter;
  height: 100vh;
}

.filter-title {
  font-weight: 600;
}

.filter-data {
  display: flex;
  flex-direction: column;
  flex-wrap: wrap;
  border-top: 2px solid #012025;
}

.filter-item {
  margin: 8px 16px;
}

.filter-checkbox {
  margin-left: 10px;
}

You'll notice as you look through the CSS that I have two distinct sets of classes in here:

Ones that are product based
And ones that are filter based.

It should be fairly self-explanatory what each set of classes is responsible for styling in the JSX; aside from the product-list-container that styles the whole page (and lays out our overall grid), product classes are for the product cards displayed on the page.

<figcaption>An example of the page title and product cards displayed on the product list page</figcaption>

</figure>

And filter classes are for the filters on the left hand side.

<figcaption>The filters to filter down the products displayed on the page</figcaption>

</figure>

Let's talk about some of the individual CSS classes in the code snippet now.

product-list-container

.product-list-container {
  display: grid;
  grid-template-rows: 55px calc(100vh - 55px);
  grid-template-columns: 220px auto;
  grid-template-areas:
    'filter header'
    'filter products';
}

The product-list-container CSS at the beginning of the file is the first thing we need to focus on here because it's where the CSS Grid is defined and laid out.

display: grid: Just like when using CSS Flexbox, the first thing we must do to let our CSS know we intend to use Grid instead of display: flex or display: block, is set the display property to grid
grid-template-rows: 55px calc(100vh - 55px): Next, we define the rows we want our grid to have.
- If you look at my page's layout below the nav bar which is always present, there's the page header that says "My Products" and then the body of the rows of product cards, so in our case there really only needs to be two rows: one for the header and another for the list of products.
- Looking at the code, I gave the page title 55px of room and then the cards the remaining portion of the viewport to the cards by using the CSS calc function to subtract the 55px from whole height of the viewport: 100vh.
grid-template-columns: 220px auto: As with defining rows, CSS Grid also allows us to define columns, and since this page has a side bar and a main content portion, 2 columns should do just fine. The sidebar portion will be 220px wide and the rest of the space will be taken up by the products, so auto will suffice here.
grid-template-areas: 'filter header' 'filter products': This last line is an interesting one. While there's multiple ways to lay out CSS Grid, one of the handiest features is the ability to use grid-template-areas which let you name grid areas and then lay them out according to those names.
- Oftentimes grid-row and grid-column will suffice to place things where they belong on the grid, but in this instance, grid-template-areas makes it really easy to lay out everything exactly as it should be on the page.

As you can see from looking at the code, the grid element named filter will take up both grid rows, and the first grid column (the one that is 220px wide).

The grid element header will take up just the first row of the grid (the 55px high row), and the second column, and the grid element products will take up the second row and second column of the grid (all the remaining space, basically).

And now that the CSS grid is laid out and its columns and rows are defined, all that's left is to name the classes to match the grid elements defined.

product-list-header

.product-list-header {
  display: flex;
  align-items: center;
  justify-content: center;
  grid-area: header;
}

We'll just work our way down the CSS, I think that's easiest here, and the first class we'll focus on after product-list-container is product-list-header. The name should give you a hint of what this class is for: it wraps the "My Products" title of the page.

In addition to a little CSS Flexbox magic to align the "My Products" page title both horizontally and vertically within its space on the page, it also gets assigned a grid area.

grid-area: header: The grid-area is the property that tells an element where it should live within the parent grid container, so by giving this element the named grid area of header, it knows it should fit into the header space as defined in the grid-template-area in the product-list-container class.

products-container

.products-container {
  grid-area: products;
  overflow: auto;
}

The next class we encounter on the way down the CSS file is the products-container class. This element only needs two CSS properties:

grid-area: products: Once more, we assign this element the grid-area name of products, telling it that it should occupy all the space in the grid defined as products.
overflow: auto: We also need to add the property of overflow: auto so that the list of products will scroll independently of the other elements on the page: the header and the sidebar.

overflow: auto; is a key ingredient to our fixed sidebar remaining fixed.

Without it, if there's more products than can be displayed at once on screen, the products will still wrap and take up all the space they need but the whole grid will scroll to display them (including scrolling beyond the space the sidebar occupies). No bueno.

Just don't forget to add this property to the products-container.

filter-wrapper

.filter-wrapper {
  padding: 10px;
  background-color: #ffcfae;
  grid-area: filter;
  height: 100vh;
}

Then we have to skip down through a few classes that exist to organize the products so they lay out nicely in rows on the page, regardless of the amount of items, until we come to the filter-wrapper class.

There's some standard padding, and background-color CSS properties at work, but the two properties to focus on are:

grid-area: filter: Our final grid-area designation has arrived. This class gets the filter property, so it will know to fill up the left hand side of the screen for both grid rows.
height: 100vh: The viewport height (vh) unit makes another appearance for this element. Setting the filter element's height to 100vh ensures it will always go down to the bottom of the page (and look like a true sidebar) no matter how many actual filters are available or how far down the product list a user scrolls.

Then after this class there's a few additional ones to lay out all the filter checkboxes in a nice vertical row with some decent spacing between them.

Again, not as important to this particular post as the four classes highlighted above.

And that's it: a fixed sidebar with other elements on the page that can scroll freely.

Conclusion

Building a sample application for users taking my course about how to modernize React applications helped me learn some cool new web development techniques along the way, not the least of which was how to build a page with a static sidebar and a scrolling center section using CSS Grid.

While this tutorial used CSS Grid to build a fixed sidebar, it could be applied to so many different situations that call for something similar: a table of contents for a blog post, a set of store details alongside a map of their locations, food items already in a cart as a user adds more to the list - the possibilities go on.

Thanks for reading. I hope you enjoyed seeing how just a few CSS Grid commands can eliminate the need to break the DOM flow with CSS like position: fixed and still give fine grained control. It takes a little practice to get the hang of CSS Grid, but once you do, I guarantee you won't want to go back. I can't imagine doing web development today without CSS Flexbox and CSS Grid at my fingertips.

If you enjoyed this post, you may also be interested in checking out my full course "The newline Guide to Modernizing an Enterprise React App".

In 10 modules and 54 lessons, I cover all the things I learned while building applications at The Home Depot. I deep dive into building and maintaining large, mission-critical React applications - because at big companies like this, it's about so much more than just making the code work.

From tooling and refactoring, to testing and design system libraries, there's a ton of material and hands-on practice here to prepare any React developer to build software that lives up to today's high standards. I hope you'll check it out.

References & Further Resources

CSS Grid Mozilla docs
CSS-Tricks: A Complete Guide to Grid
The newline Guide to Modernizing an Enterprise React App course
CSS Flexbox Mozilla docs
CSS-Tricks: A Complete Guide to Flexbox

Customize and Style Complex Data in React Table

Mon, 04 Apr 2022 00:00:00 GMT

import YouTube from '../../src/components/YouTube.astro';

</figure>

Introduction

This past summer, I started working for an Internet of Things (IoT) startup, Blues Wireless, that aims to make IoT development simpler - even when reliable Internet connections are not available. Blues does this via Notecards - prepaid cellular devices that can be embedded into any IoT device "on the edge" to transmit sensor data as JSON to a secure cloud: Notehub.

Frontend web development is what I specialize in, not hardware or firmware development, so as I get more familiar with IoT development, I started by building a simpler project: an asset tracker using a Blues Notecard, Blues Notecarrier AL with a built-in GPS antenna, and a small lithium-ion polymer (LiPo) battery to power the device.

With the help of the Blues developer experience docs, I had temperature, voltage, and GPS location data being delivered to the Notehub cloud from my tracker in less than half an hour. A good start, but the way that data from sensors really becomes useful is when it's displayed to users in some sort of UI. It could be maps, charts, or in my case, tables.

So I wanted to take my data from the Notehub cloud and pump it into a custom-made dashboard to track and display the Notecard's location and data readings from the real world. As a frontend developer, React is my current JavaScript framework of choice, and to get some more Next.js experience, I decided to build a Next.js TypeScript-powered dashboard, and I learned a ton of interesting things in the process. Over the course of a series of blog posts (which I'll link to in this article), I'll share with you some of the cools things I discovered.

If you missed the first two installments of my blogs about building an asset tracking map in Next.js using React Leaflet, you can read it here.

The second one about making data visualization charts is available here.

This post will show you how to use the React Table library to make a data table and customize the data displayed in the cells, filled with real-world location data.

Here's what the final dashboard looks like - the table of location coordinates and data at the bottom is the focus for this particular post.

Create a table component in the Next.js app

NOTE: This article will not go through the initial setup of a brand new Next.js app - that's outside the scope of this blog. If you're starting from scratch, I would recommend following the Next.js starter app with TypeScript documentation.

If you'd prefer, you can also fork and download my whole, working code base from the GitHub repo here.

Install table dependencies

Let's kick off this post by adding the necessary table library to this Next.js app.

Although there's many React-focused table libraries and components to choose from, I went with the React Table library because it is a "headless" table-library that gives you a collection of lightweight, composable, and extensible hooks for building powerful tools and data grid experiences.

The "headless" part means that there's no markup or styles rendered via React Table, so you can customize and style it exactly as you want. It may sound a little odd at first, but it actually ends up working out really well as you'll see soon.

So, install React Table in the Next.js app by typing this command into the terminal.

$ npm install react-table

NOTE: React Table is compatible with React v16.8+, so please ensure your app's version of React is at least to here if not above.

TypeScript Note:

If you're using TypeScript in your project (like I am), you'll also want to install the following dev dependency to avoid TypeScript errors:

$ npm install @types/react-table --save-dev

And now, we're ready to start using React Table to make a table in the application to display location data.

Create the `<EventTable>` component and style it

<figcaption>The custom styled event table we'll be building.</figcaption>

</figure>

For my table, I want to display "events", which are what Blues Wireless's Notecards produce and send to its cloud Notehub. Each event is a separate JSON object, and although there are all sorts of different event types stored in a single project, the ones of concern today are the _track.qo events.

_track.qo event example

A typical _track.qo event looks like this:

{
    "uid": "d7cf7475-45ff-4d8c-b02a-64de9f15f538",
    "device_uid": "dev:864475ABCDEF",
    "file": "_track.qo",
    "captured": "2021-11-05T16:10:52Z",
    "received": "2021-11-05T16:11:29Z",
    "body": {
      "hdop": 3,
      "seconds": 90,
      "motion": 76,
      "temperature": 20.1875,
      "time": 1636123230,
      "voltage": 4.2578125
    },
    "gps_location": {
      "when": "2021-11-05T16:10:53Z",
      "name": "Sandy Springs, GA",
      "country": "US",
      "timezone": "America/New_York",
      "latitude": 33.913747500000014,
      "longitude": -84.35008984375
    }
  }

It contains data like temperature, time, voltage, and gps_location coordinates. Useful stuff you might want to display in a table for easy readability, right?

Right. So here's how I built a reusable table component in a new file in the Next.js app named EventTable.tsx.

I recommend copying the following code and pasting it into your own component file, and you can also click on the file title below to see the live code repo in GitHub.

EventTable

/* eslint-disable react/jsx-key */
import { usePagination, useTable, Column } from "react-table";
import styles from "../../styles/EventTable.module.scss";

const EventTable = ({
  columns,
  data,
}: {
  data: Array<any>;
  columns: Array<Column>;
}) => {
  const {
    getTableProps, // table props from react-table
    getTableBodyProps, // table body props from react-table
    headerGroups, // headerGroups, if your table has groupings
    prepareRow, // rows for the table based on the data passed
    page, // Instead of using 'rows', we'll use page
    canPreviousPage,
    canNextPage,
    pageOptions,
    pageCount,
    gotoPage,
    nextPage,
    previousPage,
    setPageSize,
    state: { pageIndex, pageSize },
  } = useTable(
    {
      columns,
      data,
    },
    usePagination
  );

  return (
    <>
      <h2>Tracker Events</h2>
      <table className={styles.tableWrapper} {...getTableProps()}>
        <thead>
          {headerGroups.map((headerGroup) => (
            <tr {...headerGroup.getHeaderGroupProps()}>
              {headerGroup.headers.map((column, _index) => (
                <th {...column.getHeaderProps()}>{column.render("Header")}</th>
              ))}
            </tr>
          ))}
        </thead>
        <tbody {...getTableBodyProps()}>
          {page.map((row) => {
            prepareRow(row);
            return (
              <tr {...row.getRowProps()}>
                {row.cells.map((cell) => {
                  return (
                    <td {...cell.getCellProps()}>{cell.render("Cell")}</td>
                  );
                })}
              </tr>
            );
          })}
        </tbody>
      </table>
      <div className="pagination">
        <button onClick={() => gotoPage(0)} disabled={!canPreviousPage}>
          {"<<"}
        </button>{" "}
        <button onClick={() => previousPage()} disabled={!canPreviousPage}>
          {"<"}
        </button>{" "}
        <button onClick={() => nextPage()} disabled={!canNextPage}>
          {">"}
        </button>{" "}
        <button onClick={() => gotoPage(pageCount - 1)} disabled={!canNextPage}>
          {">>"}
        </button>{" "}
        <span>
          Page{" "}
          <strong>
            {pageIndex + 1} of {pageOptions.length}
          </strong>{" "}
        </span>
        <span>
          | Go to page:{" "}
          <input
            type="number"
            defaultValue={pageIndex + 1}
            onChange={(e) => {
              const page = e.target.value ? Number(e.target.value) - 1 : 0;
              gotoPage(page);
            }}
            style={{ width: "100px" }}
          />
        </span>{" "}
        <select
          value={pageSize}
          onChange={(e) => {
            setPageSize(Number(e.target.value));
          }}
        >
          {[10, 20, 30, 40, 50].map((pageSize) => (
            <option key={pageSize} value={pageSize}>
              Show {pageSize}
            </option>
          ))}
        </select>
      </div>
    </>
  );
};

export default EventTable;

Let's go through all the things happening in this table component - the amount of props being destructured right off the bat can be a little overwhelming at first glance.

Although the actual import list from the react-table library is very small - just three separate methods, the depth of functionality contained within expands quickly after the component's declared.

useTable and usePagination combined make up all of the properties being destructured at the beginning of the component, which makes sense - this is a table libary we're working with, after all.

From these two hooks we get:

getTableProps - the table props from react-table.
getTableBodyProps - the table body props from react-table.
headerGroups - headerGroups, if your table has groupings.
prepareRow - rows for the table based on the data passed.
page - necessary for having a paginated table.
canPreviousPage - boolean if there are previous pages the table can paginate to.
canNextPage - boolean if there are future pages the table can paginate to.
pageOptions - an array corresponding to available pages in the table (useful for select interfaces allowing users to type in a page number instead of using buttons).
pageCount - amount of pages available based on current page size value.
gotoPage - function to set page index to value specified by user.
nextPage - function to increase page index by one.
previousPage - function to decrease page index by one.
setPageSize - function to set page size to a new value.
state: { pageIndex, pageSize } - currently set page index and page size for table.

Whew! But after all that initial destructuring the <EventTable> component only takes in two initial array props: columns and data. Note that both of these arrays of values must be memoized, according to the React Table docs.

data is what the useTable hook turns into rows and pages of event data.
columns are the core columns configuration object for the entire table (put a pin in this array - we'll get to configuring it later in this article).

And after that it's all JSX in this component.

The headerGroups are mapped over to render any headers at the very top of the table - our table will only have one header, but you could have multiple headers helping to visually show grouping of columns.

Then, each page is mapped over, each row of data in that page is extracted and each cell in that row is rendered.

Followed by buttons galore and a custom input, which are added to make pagination possible in many different ways; canPreviousPage, nextPage, and goToPage, for example.

And finally, pageSize, or the number of rows displayed in each page, and is also made dynamic.

It's a lot of JSX, but it is fairly straightforward once some of the initial mapping and iterating is out of the way.

The nice part though, is that the react-table library has all of this functionality built-in, and all we have to pass it in terms of data to get all this is two arrays. That's it.

Style the table

Now for some very simple styling to make this table presentable looking. As I mentioned before, React Table considers itself more of a headless table utility library, and as such, applies no default styling to any of the tables it generates out of the box.

Luckily, this also means there's no default styling to undo, so just a few lines of CSS can make this table look nice.

Wherever you're keeping styles in the Next.js app (I kept all of mine in the styles/ folder), create a new CSS or SCSS file named EventTable.module.css.

I'm using Sass in my Next project in addition to the built-in CSS module styling.

There's very little Sass nesting going on in this particular SCSS file so I think it will be easy enough to follow, but if you'd like to set up Sass in your own Next project, you can follow Next's directions here.

EventTable.module.scss

.tableWrapper {
  border-spacing: 0;
  border: 1px solid #ededed;
  width: 100%;

  tr:nth-child(2n) {
    background-color: #fafafa;
  }

  th {
    padding: 15px;
  }

  td {
    padding: 5px 10px;
  }
}

In this SCSS file we're making a tableWrapper class to give the table a light border (border: 1px solid #ededed;), alternating the background color of the rows between the default background color or plain white with another off-white color (background-color: #fafafa;) for better readability, and adding some padding to the table header (padding: 15px;) and table data (padding: 5px 10px;).

Not that much styling, but it makes for a polished-enough looking table that goes with the minimalist feel of the rest of the dashboard.

NOTE: A lot of this code for the <EventTable> was inspired by the very good examples in the React Table documentation, and I would strongly encourage you to check it out - it's very thorough and covers many of the table use cases people have.

Render the table in the app

With our table component done, it's time to add it to the main dashboard component. Import it into the index.tsx file in your main pages/ folder.

I've condensed down the code in this file for clarity, but for the full code in GitHub, you can click on the file title here.

The actual data and columns for the table will be handled shortly - this is just to get the <EventTable> into the dashboard page.

pages/index.tsx

// imports
import EventTable from "../src/components/EventTable";
// other imports 

type dataProps = {
// condensed for code brevity
};

export default function Home({ data }: { data: dataProps[] }) {
// logic to transform data into the columns and data arrays needed to pass to the table

  return (
    <div>
      {/* extra tracker app code */}
      <main>
        <h1>React Blues Wireless Asset Tracker</h1>
        {/* other tracker components */}
        <div>
          <EventTable columns={columns} data={eventTableData} />
        </div>
      </main>
    </div>
  );
}

// more code down here: getStaticProps

This is one of the easiest parts of this whole tutorial: import the <EventTable> component at the top of the file, and then pop it into the JSX down below.

Now on to populating this table with events data.

Fetch data for the table

In my first asset tracking dashboard post I went into great detail about how to create your own asset tracker to generate real data for the app using Blues Wireless hardware and a data fetching function.

If you'd like to follow along there to build your own tracker and route data to Notehub, please be my guest.

For this post, I'll jump ahead to the part where we're already pulling data into the app via a Next.js getStaticProps API call. The JSON data from the Notehub cloud looks like this:

[
  {
    "uid": "d7cf7475-45ff-4d8c-b02a-64de9f15f538",
    "device_uid": "dev:864475ABCDEF",
    "file": "_track.qo",
    "captured": "2021-11-05T16:10:52Z",
    "received": "2021-11-05T16:11:29Z",
    "body": {
      "hdop": 3,
      "seconds": 90,
      "motion": 76,
      "temperature": 20.1875,
      "time": 1636123230,
      "voltage": 4.2578125
    },
    "gps_location": {
      "when": "2021-11-05T16:10:53Z",
      "name": "Sandy Springs, GA",
      "country": "US",
      "timezone": "America/New_York",
      "latitude": 33.913747500000014,
      "longitude": -84.35008984375
    }
  },
  {
    "uid": "3b1ef772-44da-455a-a846-446a85a70050",
    "device_uid": "dev:864475ABCDEF",
    "file": "_track.qo",
    "captured": "2021-11-05T22:22:18Z",
    "received": "2021-11-05T22:23:12Z",
    "body": {
      "hdop": 2,
      "motion": 203,
      "seconds": 174,
      "temperature": 22,
      "time": 1636150938,
      "voltage": 4.2265625
    },
    "gps_location": {
      "when": "2021-11-05T22:22:19Z",
      "name": "Doraville, GA",
      "country": "US",
      "timezone": "America/New_York",
      "latitude": 33.901052500000006,
      "longitude": -84.27090234375
    }
  },
  {
    "uid": "e94b0c68-b1d0-49cb-8361-d622d2d0081e",
    "device_uid": "dev:864475ABCDEF",
    "file": "_track.qo",
    "captured": "2021-11-05T22:40:04Z",
    "received": "2021-11-05T22:46:30Z",
    "body": {
      "hdop": 1,
      "motion": 50,
      "seconds": 41,
      "temperature": 21.875,
      "time": 1636152004,
      "voltage": 4.1875
    },
    "gps_location": {
      "when": "2021-11-05T22:40:05Z",
      "name": "Peachtree Corners, GA",
      "country": "US",
      "timezone": "America/New_York",
      "latitude": 33.9828325,
      "longitude": -84.21591015624999
    }
  },
  {
    "uid": "1344517c-adcb-4133-af6a-b1132ffc86ea",
    "device_uid": "dev:864475ABCDEF",
    "file": "_track.qo",
    "captured": "2021-11-06T03:04:07Z",
    "received": "2021-11-06T03:10:51Z",
    "body": {
      "hdop": 1,
      "motion": 126,
      "seconds": 218,
      "temperature": 12.5625,
      "time": 1636167847,
      "voltage": 4.1875
    },
    "gps_location": {
      "when": "2021-11-06T03:04:08Z",
      "name": "Norcross, GA",
      "country": "US",
      "timezone": "America/New_York",
      "latitude": 33.937182500000006,
      "longitude": -84.25278515625
    }
  }
]

Each JSON object in this array is a separate _track.qo motion event that displays the Notecard's current location and sensor readings. The part of the object that we care about in this particular post is the gps_location.latitude, gps_location.longitude, and body.voltage values. This is the data our table needs.

If you'd like to save setting up your own asset tracker, you can paste this small set of events into your app's index.js file to simulate the data being fetched from Notehub, for testing purposes.

Transform the JSON data to fit custom table columns and cells

With data coming into our application, we'll manipulate it and create some new columns to display in the table. And this is where things start to get interesting.

Customize the cell data

If you notice from the shape of the JSON data coming in, the gps_location.latitude and gps_location.longitude values are two separate properties in the _track.qo event, but it makes more sense to display them together in a single data cell as a comma separated list.

Likewise, the time is in epoch time - a very machine readable format, that humans have a tough time deciphering. So for my table cells, it'd be nice to format it into a date that makes sense to people.

Here's how we'll do it - as always, the full code is available for you to review in GitHub.

pages/index.tsx

// imports
import { useMemo, useEffect, useState } from "react";
import dayjs from "dayjs";
// other imports

type dataProps = {
// condensed for code brevity
};

export default function Home({ data }: { data: dataProps[] }) {

// state variables for the various pieces of data passed to the table
  const [eventTableData, setEventTableData] = useState<dataProps[]>([]);

  useEffect(() => {
    if (data && data.length > 0) {
      const eventData = [...data].reverse();
      setEventTableData(eventData);
    }
  }, [data]);

  interface row {
    [row: { string }]: any;
  }

  const columns = useMemo(
    () => [
      {
        Header: "Latest Events",
        columns: [
          {
            Header: "Date",
            accessor: "captured",
            Cell: (props: { value: string }) => {
              const tidyDate = dayjs(props.value).format("MMM D, YY h:mm A");
              return <span>{tidyDate}</span>;
            },
          },
          {
            Header: "Voltage",
            accessor: "body.voltage",
            Cell: (props: { value: string }) => {
              const tidyVoltage = Number(props.value).toFixed(2);
              return <span>{tidyVoltage}V</span>;
            },
          },
          {
            Header: "Heartbeat",
            accessor: "body.status",
          },
          {
            Header: "GPS Location",
            accessor: "gps_location",
            Cell: (row) => {
              return (
                <span>
                  {row.row.original.gps_location.latitude.toFixed(6)}
                  &#176;,&nbsp;
                  {row.row.original.gps_location.longitude.toFixed(6)}&#176;
                </span>
              );
            },
          },
        ],
      },
    ],
    []
  );

  return (
    <div>
      {/* extra tracker app code */}
      <main>
        <h1>React Blues Wireless Asset Tracker</h1>
        {/* other tracker components */}
        <div>
          <EventTable columns={columns} data={eventTableData} />
        </div>
      </main>
    </div>
  );
}

// getStaticProps call to Notehub

To make this happen, we'll add in a few React Hooks: useState, useEffect and useMemo, and to make formatting the time easier, I added the library dayjs to my project. Not entirely necessary, but it's convenient.

Get the table data

Inside of the component, create a new state variable with the useState Hook to hold the event data: const [eventTableData, setEventTableData] = useState<dataProps[]>([]);.

Next, set up a useEffect function that will run when the JSON data is fetched into the component on page load via the getStaticProps call. When the data is present, we take the data, reverse the order so the most recent events are first instead of last - I think that makes more sense when seeing a list of events here: most recent first - and set those events in the component's state via setEventTableData.

Configure the columns

The final step is defining the columns for the table.

The first thing we must do is wrap the newly defined columns variable in a useMemo function (as defined by the React Table documentation), and I'll also give the whole table a header named Latest Events.

Then we can set up the columns array: I want one for the Date stamp, one to display the device's Voltage, one for the device's Status (i.e. whether the event is a motion event caused by actual motion or it's a "heartbeat" event - an event just to let you know the device is still alive), and one for the GPS Location, which as I said before I want to be a combination of latitude and longitude.

Each of these titles will become a Header property in each of these objects, and they will be followed by an accessor property. Typically, the accessor value will be something simple like body.status, which is the way to get the actual value out of the object. If the accessor, however, needs to be transformed before being set, this is where the Cell function comes into play.

Cell is a method that can be be used to transform the props in that cell and completely customize the return value, right down to the JSX.

So for example, to modify the timestamp date into a nicely formatted string, we grab the captured value out of the event, and then use Cell to run a dayjs function and format the date into a new variable named tidyDate. And then we return tidyDate as a bit of JSX wrapped in a <span> tag.

Similarly, to concatenate the latitude and longitude values together into one cell, we have to go even deeper into the props being passed to Cell, and pull out two of the properties nested inside of the gps_location object to return in the cell.

Console.log() is your friend here.

Don't be afraid to reach for console.log() when putting together these complex data cells. Only by using it to see the shape of the data and each row in the table was I able to figure out how to extract the values I needed for the GPS Location column.

And there you have it: our columns are defined and our data is too, pass both to the <EventTable> component, and we should be good to go.

Conclusion

About 9 months ago, I began working for an IoT startup, and learning how to manage and display readings in dashboards from devices "on the edge". So one of the first projects I built to this end was a dashboard including data charts, a map showing device location, and a table listing all the events being displayed here.

To render a table with all the flexibility and none of the cruft, I turned to the React Table library, a headless and highly extensible set of hooks that simplifies building complex tables, handling things like pagination, page size, and so much more, with very little effort on our part. Even modifying the display of table cell data in a row, is relatively straightforward. It's a great library.

This dashboard actually came in handy when my parents' car was stolen from their driveway the night after Thanksgiving. If you want to hear the whole story and build your own tracker, check out this blog post and video I made for Blues Wireless - it details the whole process from hardware to software to deploying to Netlify.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading. I hope you enjoyed learning how to leverage the React Table library to make a data table to display event data from an IoT asset tracker. Tables can be tricky to get right, but they're such a nice, straightforward way to display all sorts of data to users.

References & Further Resources

Hackster.io original asset tracker project
Asset tracker GitHub repo
React Table library
Blues Wireless website
Blues Wireless developer experience site
Original article about rendering maps in a Next.js application
Follow up article about custom styling charts and tooltips with React Recharts library

Build and Custom Style Recharts Data Charts

Thu, 10 Mar 2022 00:00:00 GMT

import YouTube from '../../src/components/YouTube.astro';

</figure>

Introduction

Over the summer of 2021 I started working for an Internet of Things (IoT) startup, Blues Wireless that aims to make IoT development easy - even when reliable Internet connections are not available. Blues does this via Notecards - prepaid cellular devices that can be embedded into any IoT device "on the edge" to transmit sensor data as JSON to a secure cloud: Notehub.

I am not a hardware or firmware engineer however, web development is my area of expertise (frontend in particular), so to dip my toes into IoT development, I started off building a simpler project: an asset tracker using only a Blues Notecard, Blues Notecarrier AL with a built-in GPS antenna, and a small lithium-ion polymer (LiPo) battery to power it all.

With the help of the Blues developer experience docs, I had temperature, voltage, and GPS location data being delivered to the Notehub cloud from my tracker in under 30 minutes. That's cool and all, but the way that data from sensors in the world really becomes useful is when it's displayed to users in some sort of UI, right? It could be maps, tables, or in my case, charts.

So I wanted to take my data from the Notehub cloud and pump it into a custom-made dashboard to track and display the Notecard's location, voltage, and sensor readings in the real world. React is my current JavaScript framework of choice, and I decided to build a Next.js TypeScript-powered dashboard, and I learned a ton of interesting things in the process, which I intend to share with you over a series of blog posts in the next few months.

If you missed my first blog post installment all about building an asset tracking map in Next.js using React Leaflet, you can read it here.

Today, I plan to show you how to build charts and a custom styled tooltip in a Next.js application with the Recharts Library, using real-world data from a third-party API source.

Here's what the final dashboard looks like - the temperature and voltage charts are the focus for this particular post.

Create chart components in the Next.js app

NOTE This article will not go through the initial setup of a brand new Next.js app - that's outside the scope of this blog. If you're starting from scratch, I would recommend following the Next.js starter app with TypeScript documentation.

If you'd prefer, you can also fork and download my whole, working code base from the GitHub repo here.

Install chart dependencies

To get started adding charts to our Next.js application, we have to install the Recharts library. Recharts is a chart library built with React and the popular data visualization library D3, and it makes it easy to use React-based chart components that feel familiar to make all sorts of different types of charts, while still allowing a high degree of flexibility for customization, as needed.

The API and examples documentation for Recharts are thorough, and I'd highly encourage you to read them to get a feel for the variety of charts available and how to work with them.

Install the Recharts library in the Next.js app with the following commands in a terminal.

$ npm install recharts

This is the only new library needed to get Recharts up and running, even when using TypeScript, so it's time to move on and make some various chart components.

Set up a `<TempChart >` component

The first type of chart component we'll create for the dashboard will be used to display the temperature data supplied by the Notecard. This will be an area chart that looks like the image here.

<figcaption>This is a Recharts area chart displaying temperature data over time, including a tooltip.</figcaption>

</figure>

Since this component will have the sole goal of displaying temperature data, create a new file in the repo named TempChart.tsx, and paste the following code into it. The actual code in GitHub is available by clicking the file title below.

TempChart.tsx

import {
  AreaChart,
  Area,
  XAxis,
  YAxis,
  Label,
  CartesianGrid,
  Tooltip,
  ResponsiveContainer,
} from "recharts";
import CustomTooltip from "./CustomTooltip";

type tempDataProps = {
  shortenedDate: string;
  date: string;
  temp: number;
}[];

const TempChart = ({ tempData }: { tempData: tempDataProps }) => {
  return (
    <>
      <h2>Temperature Chart</h2>
      <ResponsiveContainer width={"100%"} height={300} min-width={300}>
        <AreaChart
          data={tempData}
          margin={{
            top: 10,
            right: 30,
            left: 20,
            bottom: 30,
          }}
        >
          <CartesianGrid strokeDasharray="3 3" />
          <XAxis dataKey="shortenedDate">
            <Label value="Date" position="bottom" />
          </XAxis>
          <YAxis dataKey="temp">
            <Label
              value="Temperature &#8457;"
              angle={-90}
              position="left"
              dy="-10"
            />
          </YAxis>
          <Tooltip content={<CustomTooltip payload={tempData} />} />
          <Area
            type="monotone"
            dataKey="temp"
            stroke="#8884d8"
            fill="#8884d8"
          />
        </AreaChart>
      </ResponsiveContainer>
    </>
  );
};

export default TempChart;

There's a lot of imports from Recharts happening in this component, so let's go through them one by one to understand what their purposes are.

Each of the imports from the recharts library at the very top of the file represent a different component used to make this chart display correctly.

AreaChart is the container for all the chart data and is how Recharts knows what type of chart it's rendering.
Area is where the array of data to be displayed by the chart is passed. The stroke and fill properties in the <Area> component allow us to define the chart's line color and fill color as hex codes.
- (Note: if there was more than one array of temperature data this chart needed to display - say, for multiple Notecards reporting data, a second <Area> component would be included in the JSX underneath the first instance)
XAxis and YAxis are how the chart knows what the X and Y axes of the chart should use as their data displays.
- The dataKey property inside of the axis component is the property name of the array data being passed to the chart. For example, each item in the array of data being passed to the temperature chart's <YAxis> has a property of "temp", so that is the dataKey for the <YAxis> component.
Label must always be a child component of the XAxis or YAxis component and it displays a custom axis label.
- I like it because the chart label, label angle, value, positioning, and more can be changed very easily here.
CartesianGrid draws the nice squares which align with the cartesian coordinate axes.
- This is not a required component, but I think it improves chart readability. The strokeDasharray property is customizable, so feel free to play with the numbers to find a square pattern that suits you.
Tooltip although our tooltip will end up being a custom component, in order to display it the custom one, we must pass it into the generic <Tooltip> component as the property content. We'll get into this more when we create the tooltip component, so put a pin in this for now.
ResponsiveContainer wraps the whole chart component and makes it responsive to the size of the parent container. Although the chart component itself can be given a height and width, they're static. When passing the <ResponsiveContainer> component width and height props, one should be a percentage string.
- For my component, the width is width={100%}, for this reason.

The only other import to this file is the <CustomTooltip> component and we'll get to this component in depth soon.

After all the imports, the tempDataProps are defined for the TypeScript array of data being passed to this chart. The props are pretty simple: each object in the array will contain:

A shortenedDate (this is a simplified date format of MM/DD/YYYY) to display in the <XAxis> component,
A date - the longer form date of MMM D, YYYY h:mm A to display in the tooltip,
And a temp - the actual temperature.

Finally, all of these chart components are composed in the JSX to make a responsive area chart with dates on the X axis, temperature on the Y axis, a lavender stroke and fill color, and a custom tooltip.

Build a `<VoltageChart>` component

The second chart component we're going to build is for the Notecard's voltage levels over time. The Notecard is designed to be ultra low power so even small batteries can power it for long stretches of time (multiple years, even), but it's still good to keep an eye on the voltage so if it does start to drop, the LiPo battery can be recharged or swapped out.

<figcaption>This is a Recharts line chart displaying voltage data over time, including a tooltip.</figcaption>

</figure>

This component will be a standard line chart instead of an area chart, so create a new component file named VoltageChart.tsx in your project.

A lot of the code for the <VoltageChart> component should feel familiar when you read over it - it's quite similar to the <TempChart> component's code. As before, you can click the file title to go the GitHub code.

Yes, this chart code isn't the DRYest (don't repeat yourself).

If this was a larger project with more charts, I would work to make some more generic, "base chart" components to be more flexible for various purposes, but since it's a relatively small app with just a two charts, I kept it simpler.

VoltageChart.tsx

import {
  LineChart,
  Line,
  XAxis,
  YAxis,
  Label,
  CartesianGrid,
  Tooltip,
  ResponsiveContainer,
} from "recharts";
import CustomTooltip from "./CustomTooltip";

type voltageDataChart = {
  shortenedDate: string;
  date: string;
  voltage: number;
}[];

const VoltageChart = ({ voltageData }: { voltageData: voltageDataChart }) => {
  return (
    <>
      <h2>Voltage Chart</h2>
      <ResponsiveContainer width={"100%"} height={300} min-width={300}>
        <LineChart
          data={voltageData}
          margin={{
            top: 10,
            right: 30,
            left: 20,
            bottom: 30,
          }}
        >
          <CartesianGrid strokeDasharray="3 3" />
          <XAxis dataKey="shortenedDate">
            <Label value="Date" position="bottom" />
          </XAxis>
          <YAxis dataKey="voltage" type="number" domain={[3.3, 5.6]}>
            <Label value="Voltage V" angle={-90} position="left" dy="-10" />
          </YAxis>
          <Tooltip content={<CustomTooltip payload={voltageData} />} />
          <Line type="monotone" dataKey="voltage" stroke="#82ca9d" />
        </LineChart>
      </ResponsiveContainer>
    </>
  );
};

export default VoltageChart;

As with the <TempChart>, the <VoltageChart> imports all the same Recharts components except it uses LineChart and Line to indicate that this chart will be displayed as a line chart instead of an area chart and tell the <Line> component the type will be an unbroken monotone type with a soft green stroke color.

Beyond these key differences everything else is much the same: same <ResponsiveContainer> chart wrapper, similarly shaped voltageData array (the temp property on each object is replaced by voltage here), same X and Y axis data (although this Y axis has a domain property that sets the upper and lower bounds of the chart), and even the same <CustomTooltip> component.

So now that we've defined our charts, it's time to dive into that <CustomTooltip> component.

Make a `<CustomTooltip>` component and style it

This is where things stopped being so straightforward. If we examine the Recharts tooltip API documentation closely, you'll see mention of the method content as a way to render a custom tooltip, lucky for us, this method links to example code so we can see what props a custom tooltip component can take, including CSS styling.

Here is what our custom tooltip is going to end up looking like.

<figcaption>A custom-styled tooltip in Recharts.</figcaption>

</figure>

In the app, make a new component file named CustomTooltip.tsx, and go ahead and make a new style file named CustomTooltip.modules.scss.

I'm using Sass in my Next project in addition to the built-in CSS module styling.

There's very little Sass nesting going on in this particular SCSS file so I think it will be easy enough to follow, but if you'd like to set up Sass in your own Next project, you can follow Next's directions here.

CustomTooltip.tsx

import styles from "../../styles/CustomTooltip.module.scss";

const CustomTooltip = ({ active, payload }: any) => {
  if (active && payload && payload.length) {
    return (
      <div className={styles.customTooltip}>
        <div className={styles.tooltipDetails}>
          <p className={styles.label}>Date: {payload[0].payload.date}</p>
          {payload[0].payload.temp ? (
            <p>Temp {payload[0].payload.temp}&#8457;</p>
          ) : (
            <p>Voltage {payload[0].payload.voltage}V</p>
          )}
        </div>
      </div>
    );
  }

  return null;
};

export default CustomTooltip;

The CustomTooltip component here takes in two props:

active - a boolean denoting if a tooltip should be displayed (this happens when a user mouses over the chart on desktop or clicks on a particular point on mobile),
payload - the data the tooltip will be displaying from the chart. For us, we want the chart's temp (or voltage) and the more specific date timestamp in our tooltip.

The custom tooltip's JSX checks if active is true for a particular data point and if the payload exists as a single object in the array. If both conditions are met, the tooltip displays the longer format date and either the temperature or voltage depending on which chart the tooltip's being invoked with.

Once again, for a larger project the display would need to be redesigned to be more flexible to whatever data props were being passed in.

Our data's being displayed in the tooltip, now to give it a little styling to look decent.

CustomTooltip.scss

.customTooltip {
  background: white;
  color: #333;
  box-shadow: 0 3px 14px rgb(0 0 0 / 40%);
  padding: 1px;
  text-align: left;
  border-radius: 12px;

  .tooltipDetails {
    margin: 13px 19px;
  }

  .label {
    font-weight: 600;
  }
}

The customTooltip class wraps all the tooltip data, giving it a white background, black text with a little padding, a slight box shadow, and some nicely rounded corners. Nested inside of this class are tooltipDetails which defines some margins around the text, and label, which gives extra emphasis to the font.

Then import the SCSS file into the CustomTooltip component file, define the module's name as styles and plug in the corresponding CSS in the correct components in the JSX. Now when this component is imported into the two chart components, it should look good and display the more specific tooltip data we want.

With that done, we're ready to start using these components in the dashboard page.

Render the charts in the application

Ok, so in the main file where all our dashboard components are imported, let's add our charts and prepare to pass data to them.

pages/index.tsx

// imports
import TempChart from "../src/components/TempChart";
import VoltageChart from "../src/components/VoltageChart";
// other imports

type dataProps = {
// condensed for code brevity
};

export default function Home({ data }: { data: dataProps[] }) {
// logic to transform data into the arrays needed to pass to the charts

return (
    <div>
      {/* extra tracker app code */}
      <main>
        <h1>React Blues Wireless Asset Tracker</h1>
        <div>
          <TempChart tempData={tempData} />
        </div>
        {/* other tracker components */}
        <div>
          <VoltageChart voltageData={voltageData} />
        </div>
        {/* other tracker components */}
      </main>
    </div>
  );
}

// more code down here: getStaticProps that fetches the data

There's not much to these imports, they're standard default imports to get these components into the file. After the two chart components are imported, they're rendered inside of the JSX down at the bottom of the file. The getStaticProps function at the bottom of the file that I've written a comment for is where the Notehub data gets pulled into the app, and the comment right under our export default function line is where we'll transform that raw data into the tempData and voltageData arrays that our charts need.

Let's get to it.

Pull in data for the charts

If you'd like to follow along there to build your own tracker and route data to Notehub, please be my guest.

For this post, I'll jump ahead to the part where we're already pulling data into the app via a Next.js getStaticProps API call. The JSON data from the Notehub cloud looks like this:

[
  {
    "uid": "d7cf7475-45ff-4d8c-b02a-64de9f15f538",
    "device_uid": "dev:864475ABCDEF",
    "file": "_track.qo",
    "captured": "2021-11-05T16:10:52Z",
    "received": "2021-11-05T16:11:29Z",
    "body": {
      "hdop": 3,
      "seconds": 90,
      "motion": 76,
      "temperature": 20.1875,
      "time": 1636123230,
      "voltage": 4.2578125
    },
    "gps_location": {
      "when": "2021-11-05T16:10:53Z",
      "name": "Sandy Springs, GA",
      "country": "US",
      "timezone": "America/New_York",
      "latitude": 33.913747500000014,
      "longitude": -84.35008984375
    }
  },
  {
    "uid": "3b1ef772-44da-455a-a846-446a85a70050",
    "device_uid": "dev:864475ABCDEF",
    "file": "_track.qo",
    "captured": "2021-11-05T22:22:18Z",
    "received": "2021-11-05T22:23:12Z",
    "body": {
      "hdop": 2,
      "motion": 203,
      "seconds": 174,
      "temperature": 22,
      "time": 1636150938,
      "voltage": 4.2265625
    },
    "gps_location": {
      "when": "2021-11-05T22:22:19Z",
      "name": "Doraville, GA",
      "country": "US",
      "timezone": "America/New_York",
      "latitude": 33.901052500000006,
      "longitude": -84.27090234375
    }
  },
  {
    "uid": "e94b0c68-b1d0-49cb-8361-d622d2d0081e",
    "device_uid": "dev:864475ABCDEF",
    "file": "_track.qo",
    "captured": "2021-11-05T22:40:04Z",
    "received": "2021-11-05T22:46:30Z",
    "body": {
      "hdop": 1,
      "motion": 50,
      "seconds": 41,
      "temperature": 21.875,
      "time": 1636152004,
      "voltage": 4.1875
    },
    "gps_location": {
      "when": "2021-11-05T22:40:05Z",
      "name": "Peachtree Corners, GA",
      "country": "US",
      "timezone": "America/New_York",
      "latitude": 33.9828325,
      "longitude": -84.21591015624999
    }
  },
  {
    "uid": "1344517c-adcb-4133-af6a-b1132ffc86ea",
    "device_uid": "dev:864475ABCDEF",
    "file": "_track.qo",
    "captured": "2021-11-06T03:04:07Z",
    "received": "2021-11-06T03:10:51Z",
    "body": {
      "hdop": 1,
      "motion": 126,
      "seconds": 218,
      "temperature": 12.5625,
      "time": 1636167847,
      "voltage": 4.1875
    },
    "gps_location": {
      "when": "2021-11-06T03:04:08Z",
      "name": "Norcross, GA",
      "country": "US",
      "timezone": "America/New_York",
      "latitude": 33.937182500000006,
      "longitude": -84.25278515625
    }
  }
]

Each JSON object in this array is a separate _track.qo motion event that displays the Notecard's current location and sensor readings. The part of the object that we care about in this particular post is the body.temperature and body.voltage values. This is the data our charts need.

If you'd like to save setting up your own asset tracker, you can paste this small set of events into your app's index.js file to simulate the data being fetched from Notehub, for testing purposes.

Take that data and reshape it to work with our chart and tooltip components

With the correct shape of our data being fed into our app (either via getStaticProps and Notehub) or the JSON test data I shared above, it's time to reshape this data.

I've condensed down the logic to make this file easier to read, but you can see the full file on GitHub.

pages/index.tsx

// imports
import { useState } from "react";
import dayjs from "dayjs";
import { convertCelsiusToFahrenheit } from "../src/util/helpers";
// other imports

type dataProps = {
// condensed for code brevity
};

export default function Home({ data }: { data: dataProps[] }) {

// state variables for the various pieces of data passed to the charts
  const [tempData, setTempData] = useState<
    { date: string; shortenedDate: string; temp: number }[]
  >([]);

  const [voltageData, setVoltageData] = useState<
    { date: string; shortenedDate: string; voltage: number }[]
  >([]);

  // logic to transform data into the items needed to pass to the charts
  useEffect(() => {
    const temperatureDataArray: {
        date: string;
        shortenedDate: string;
        temp: number;
        }[] = [];
    const voltageDataArray: {
        date: string;
        shortenedDate: string;
        voltage: number;
        }[] = [];
   
    if (data && data.length > 0) {
      data
        .sort((a, b) => {
          return Number(a.captured) - Number(b.captured);
        })
        .map((event) => {
          const temperatureObj = {
            date: dayjs(event.captured).format("MMM D, YYYY h:mm A"),
            shortenedDate: dayjs(event.captured).format("MM/DD/YYYY"),
            temp: Number(convertCelsiusToFahrenheit(event.body.temperature)),
          };
          temperatureDataArray.push(temperatureObj);
          const voltageObj = {
            date: dayjs(event.captured).format("MMM D, YYYY h:mm A"),
            shortenedDate: dayjs(event.captured).format("MM/DD/YYYY"),
            voltage: Number(event.body.voltage.toFixed(2)),
          };
          voltageDataArray.push(voltageObj);
        });
    }
    setTempData(temperatureDataArray);
    setVoltageData(voltageDataArray);
  }, [data]);

return (
    <div>
      {/* extra tracker app code */}
      <main>
        <h1>React Blues Wireless Asset Tracker</h1>
        <div>
          <TempChart tempData={tempData} />
        </div>
        {/* other tracker components */}
        <div>
          <VoltageChart voltageData={voltageData} />
        </div>
        {/* other tracker components */}
      </main>
    </div>
  );
}

// getStaticProps call to Notehub

Here's what's happening in this file, we declare two new React state variables: tempData and voltageData - these arrays are what get passed to the chart components. Each of these variables is an object with three properties: date, shortenedDate, and either temp or voltage.

Then, inside of a useEffect function that runs as soon as the data variable is available via the getStaticProps call, we sort the data, and then iterate over each object in the original data array and create a new temperatureObj and voltageObj from the data that gets pushed into the temperatureDataArray and voltageDataArray variables declared at the very top of the useEffect.

Finally, those arrays are set into the component's state with our useState variables. At this point, you should have a couple of nice looking charts displaying a few points of data about your Notecard's voltage and temperature.

If you look closely, you can see a couple of extra imports at the top of this code example for the dayjs library and a helper function I named convertCelsiusToFahrenheit. The dayjs library helps make date time formatting significantly easier for our date and shortenedDate properties for our new data arrays and convertCelsiusToFahrenheit takes the Notecard's temp (reported in Celsius) and makes it Fahrenheit. Neither of these things is complicated, but I wanted to point them out for the very observant among you.

Conclusion

Once I started working for an IoT startup, getting more familiar with the possibilities of what sensors could report from the real world became more of a focus for me - but still from a web development perspective. To that end, I built a web dashboard pulling data from our hardware devices out in the field: location, temperature, battery voltage, etc.

For the data like temperature and voltage, charts seemed most appropriate to display the changing values over time, and so I chose the Recharts library to help with these data visualizations. The library's easy to work with and produces some nifty looking charts without too much effort on my part, and when I wanted to style my own tooltip and display some custom data, that was straightforward too.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading. I hope you enjoyed learning how to create a couple of charts and custom tooltips using the Recharts library - slick data visualizations have so many potential applications for transforming lots of data points into meaningful data people can draw conclusions from.

References & Further Resources

Hackster.io original asset tracker project
Asset tracker GitHub repo
Recharts chart library
Blues Wireless website
Blues Wireless developer experience site
Previous article about rendering maps in a Next.js application

How to Unit Test Next.js API Routes with TypeScript

Wed, 16 Feb 2022 00:00:00 GMT

</figure>

Introduction

Next.js is an awesome frontend framework. It's powered by React under the hood so it plays well with everything React has to offer out of the box: Hooks, Context, hot browser reloading, TypeScript integration, and then it takes it a step further than what Create React App has, and offers even more like routing, server side rendering (SSR), static site generation (SSG), all the SEO juice that comes along with both SSR and SSG, and built-in API routing - no extra Node server required to proxy API calls securely to a database, another microservice, or a third party API.

At work, a team of developers and I have been building a new application that we've open sourced to help our users get up and running faster with the Internet of Things (IoT) hardware we create.

For our first "accelerator application", the idea is that a user will get some of our IoT devices, those devices will begin collecting data like temperature, humidity, motion, etc., they'll send that environmental data to a cloud, and then they'll fork our "starter application" code to get a dashboard up and running, pulling in their own sensor data from the cloud, and displaying it in the browser.

To build this app, we decided to go with the Next.js framework because it offered so many of the benefits I listed above, one of the most important being the the ability to make secure API calls without having to set up a standalone Node server using Next.js's API routes. All of the data displayed by the application must be fetched from the cloud (or a database) where the device data is stored after it's first recorded.

And this being a production-ready application, things like automated unit and end-to-end tests to ensure the various pieces of the application work as expected are a requirement - both to give the developers and our users confidence that as new features are added already existing functionality remains intact.

While by and large, the Next.js documentation is great, one place that it does fall short is when it comes to unit testing these API routes. There is literally nothing in the documentation that touches on how to test API routes with Jest and React Testing Library - the de facto unit testing library combo when it comes to any React-based app.

Which is why today I'll be showing you how to unit test Next.js API routes, including gotchas like local environment variables, mocked data objects, and even TypeScript types for Next-specific objects like NextApiRequest.

The actual Next.js API route to test

So before we get to the tests, let me give you a brief example of the sorts of API calls this application might make. For our app, the first thing that must be fetched from the cloud is info about the "gateway devices".

NOTE: The files I've linked to in the actual repo are historical links in GitHub. The project underwent a major refactor afterwards to more cleanly divide up different layers for future ease of use and flexibility, but if you dig back far enough in the commit history (or just click on the hyperlinked file name) you can see our working code that matches what I'm describing below.

Fetch the gateway device info

The gateways are the brains of the operation - there are a number of sensors that all communicate with the gateways telling them what environmental readings they're getting at their various locations, and the gateways are responsible for sending that data from each sensor to the cloud - it's like a hub and spoke system you'd see on a bicycle wheel.

Before anything else can happen in the app, we have to get the gateway information, which can later be used to figure out which sensors and readings go with which gateways. I won't go into more details about how the app works because it's outside the scope of this post, but you can see the whole repo in GitHub here.

Let's focus on the API call going from the Next.js app to our cloud (which happens to be called Notehub). In order to query Notehub we'll need:

An authorization token,
A Notehub project's ID,
And a gateway device's ID.

Below is an example of the call made to Notehub via Next.js to fetch the gateway device data. I'll break down what's happening after the code block.

Click the file name if you'd like to see the original code this file was modeled after.

pages/api/gateways/[gatewayID].ts

import type { NextApiRequest, NextApiResponse } from 'next';
import axios, { AxiosResponse } from 'axios';

export default async function gatewaysHandler(
  req: NextApiRequest,
  res: NextApiResponse,
) {
  // Only allow GET requests
  if (req.method !== 'GET') {
    res.status(405).json({ err: 'Method not allowed' });
    return;
  }

  // Gateway UID must be a string
  if (typeof req.query.gatewayID !== 'string') {
    res.status(400).json({ err: 'Invalid gateway ID' });
    return;
  }

  // Query params
  const { gatewayID } = req.query;
  // Notehub values
  const { BASE_URL, AUTH_TOKEN, APP_ID } = process.env;
  // API path
  const endpoint = `${BASE_URL}/v1/projects/${APP_ID}/devices/${gatewayID}`;
  // API headers
  const headers = {
    'Content-Type': 'application/json',
    'X-SESSION-TOKEN': AUTH_TOKEN,
  };

  // API call
  try {
    const response: AxiosResponse = await axios.get(endpoint, { headers });
    // Return JSON
    res.status(200).json(response.data);
  } catch (err) {
    // Check if we got a useful response
    if (axios.isAxiosError(err)) {
      if (err.response && err.response.status === 404) {
        // Return 404 error
        res.status(404).json({ err: 'Unable to find device' });
      }
    } else {
      // Return 500 error
      res.status(500).json({ err: 'Failed to fetch Gateway data' });
    }
  }
}

In our code, the Axios HTTP library is used to make our HTTP requests cleaner and simpler, there are environment variables passed in from a .env.local file for various pieces of the call to the Notehub project which need to be kept secret (things like APP_ID and AUTH_TOKEN), and since this project is written in TypeScript, the NextApiRequest and NextApiResponse types also need to be imported at the top of the file.

After the imports, there's a few validation checks to make sure that the HTTP request is a GET, and the gatewayID from the query params is a string (which it always should be, but it never hurts to confirm), then the URL request to the Notehub project is constructed (endpoint) along with the required headers to allow for access, and the call is finally made with Axios. Once the JSON payload is returned from Notehub, it's read for further errors like the gateway ID cannot be found, and if everything's in order, all the gateway info is returned.

There's just enough functionality and possible error scenarios to make it interesting, but no so much that it's overwhelming to test. Time to get on with writing unit tests.

Set up API testing in Next.js

Ok, now that we've seen the actual API route we want to write unit tests for, it's time to get started. Since we're just testing API calls instead of components being rendered in the DOM, Jest is the only testing framework we'll need this time, but that being said, there's still a little extra configuration to take care of.

Install the `node-mocks-http` Library

The first thing that we'll need to do in order to mock the HTTP requests and response objects for Notehub (instead of using actual production data, which is much harder to set up correctly every time) is to install the node-mocks-http.

This library allows for mocking HTTP requests by any Node-based application that uses request and response objects (which Next.js does). It has this handy function called createMocks(), which merges together two of its other functions createRequest() and createResponse() that allow us to mock both req and res objects in the same function. This lets us dictate what Notehub should accept and return when the gatewayHandler() function is called in our tests.

Add this library to the project's devDependencies list in the package.json file like so.

npm install --save-dev node-mocks-http

Add an `.env.test.local` file for test-related environment variables

I learned the hard way that environment variables present in a Next.js project's .env.local file (the prescribed way Next wants to read environment variables) do not automatically populate to its unit tests.

Instead, we need to make a new file at the root of the project named .env.test.local to hold the test environment variables.

This file will basically be a duplicate of the env.local file.

We'll include the BASE_URL to reach our API, a valid AUTH_TOKEN, a valid APP_ID and a valid DEVICE_ID. The DEVICE_ID is the gateway device's ID, which actually comes from the app's URL query parameters but since this is just unit testing this route file's functionality, to keep all our variables in one centralized place, we'll pass the gateway's ID as an environment variable.

Here's what your test environment variables file should contain.

NOTE: Neither this file nor your actual .env.local file should ever be committed to your repo in GitHub. Make sure these are in your .gitignore file so they don't accidentally make it there where anyone could read potentially secret variables.

.env.test.local

BASE_URL=https://api.notefile.net
AUTH_TOKEN=[MY_AUTH_TOKEN]
APP_ID=[app:MY_NOTEHUB_PROJECT_ID]
DEVICE_ID=[dev:MY_GATEWAY_DEVICE_ID]

Use real env vars for your test file: Although in our final test file you won't see us importing all of these variables to construct the Notehub URL, if they are not valid and included now, the tests will error out - the tests are actually constructing valid URLs under the hood, we're just specifying what to send the receive back when the calls are placed. Undefined variables or nonsense test data variables will cause the tests to fail.

And with those two things done, we can get to testing.

Write the API tests

For keeping things in line with what Jest recommends, we can store all our test files inside of a folder at the root of the Next project named __tests__/, and to make it easy to figure out which tests go with which components, I tend to like to mimic the original file path and name for the file being tested.

If you prefer to keep your tests inline with your actual source files, that's a valid choice as well. I've worked with both kinds of code repos so it's really a matter of personal preference.

In that case, I tend to just create __tests__/ folders at the root of each folder alongside where the actual files live. So inside of the pages/api/ folder I'd make a new folder named __tests__/ and add any related test files in there.

Since this is a route API file buried within our pages/ folder, I'd recommend a similar file path inside the __tests__/ folder: __tests__/pages/api/gateways/[gatewayID].test.ts. In this way, a quick glance at the file name should tell us exactly what this file is testing.

Then, we come up with possible test cases to cover.

Some scenarios to test include:

Testing a valid response from Notehub with a valid authToken, APP_ID and DEVICE_ID which results in a 200 status code.
Testing that an invalid gateway ID for a device that doesn't exist and throws a 404 error.
Testing that no gateway ID results in a 400 error.
And testing that trying to make any type of HTTP call besides a GET results in a 405 error.

Below is what my tests look like to test this API endpoint. We'll dig into the details after the big code block.

Click the file name if you'd like to see the original code this file was modeled after.

__tests__/pages/api/gateways/[gatewayUID].test.ts

/**
 * @jest-environment node
 */
import { createMocks, RequestMethod } from 'node-mocks-http';
import type { NextApiRequest, NextApiResponse } from 'next';
import gatewaysHandler from '../../../../../src/pages/api/gateways/[gatewayUID]';

describe('/api/gateways/[gatewayUID] API Endpoint', () => {
  const authToken = process.env.AUTH_TOKEN;
  const gatewayID = process.env.DEVICE_ID;

  function mockRequestResponse(method: RequestMethod = 'GET') {
    const {
      req,
      res,
    }: { req: NextApiRequest; res: NextApiResponse } = createMocks({ method });
    req.headers = {
      'Content-Type': 'application/json',
      'X-SESSION-TOKEN': authToken,
    };
    req.query = { gatewayID: `${gatewayID}` };
    return { req, res };
  }

  it('should return a successful response from Notehub', async () => {
    const { req, res } = mockRequestResponse();
    await gatewaysHandler(req, res);

    expect(res.statusCode).toBe(200);
    expect(res.getHeaders()).toEqual({ 'content-type': 'application/json' });
    expect(res.statusMessage).toEqual('OK');
  });

  it('should return a 404 if Gateway UID is invalid', async () => {
    const { req, res } = mockRequestResponse();
    req.query = { gatewayID: 'hello_world' }; // invalid gateway ID

    await gatewaysHandler(req, res);

    expect(res.statusCode).toBe(404);
    expect(res._getJSONData()).toEqual({ err: 'Unable to find device' });
  });

  it('should return a 400 if Gateway ID is missing', async () => {
    const { req, res } = mockRequestResponse();
    req.query = {}; // Equivalent to a null gateway ID

    await gatewaysHandler(req, res);

    expect(res.statusCode).toBe(400);
    expect(res._getJSONData()).toEqual({
      err: 'Invalid gateway UID parameter',
    });
  });

  it('should return a 405 if HTTP method is not GET', async () => {
    const { req, res } = mockRequestResponse('POST'); // Invalid HTTP call

    await gatewaysHandler(req, res);

    expect(res.statusCode).toBe(405);
    expect(res._getJSONData()).toEqual({
      err: 'Method not allowed',
    });
  });
});

Handle the imports

Before writing our tests we need to import the createMocks and RequestMethod variables from the node-mocks-http library. As I noted earlier, createMocks() allows us to mock both the req and res objects in one function, instead of having to mock them separately.

Additionally, since this is a TypeScript file, we'll need to import the NextApiRequest and NextApiResponse types from next - just like for the real API route file.

And finally, we need to import the real gatewayHandler function - it's what we're trying to unit test after all.

Create a reusable mockRequestResponse() helper function

After creating a describe block to house all the unit tests, I created a reusable helper function to set up the mocked API call for each test.

This reusable mockRequestResponse() function, allows us to only have to construct our mocked HTTP call once, cuts down on the amount of duplicate code in the test files, and makes overall readability easier. Although we may change various parts of the req or res object based on what scenario is being tested, writing this function once and being able to call it inside of each test is a big code (and time) saver.

const authToken = process.env.AUTH_TOKEN;
const gatewayID = process.env.DEVICE_ID;

function mockRequestResponse(method: RequestMethod = 'GET') {
  const {
    req,
    res,
  }: { req: NextApiRequest; res: NextApiResponse } = createMocks({ method });
  req.headers = {
    'Content-Type': 'application/json',
    'X-SESSION-TOKEN': authToken,
  };
  req.query = { gatewayID: `${gatewayID}` };
  return { req, res };
}

Above, I've pulled out a snippet from the larger code block that focuses just on the mockRequestResponse() function and the two environment variables it needs to during its construction authToken and gatewayID. After declaring the function name we specify its method using the node-http-mocks RequestMethod object: method:RequestMethod="GET", and then we destructure and set the req and res object types that come from the createMocks() function as NextApiRequest and NextApiResponse (just like in our real code).

We create the same req.headers object that Notehub requires with our test-version authToken, and set the mocked query parameter gatewayID equal to the gatewayID being supplied by our .env.test.local file.

Write each test

With our mockRequestResponse() function built, we can simply call it inside of each test to get our mocked req and res objects, call the actual gatewayHandler() function with those mocked objects, and make sure the responses that come back are what we expect.

If a property on the req object needs to be modified before the call to gatewayHandler is made, it's as straight forward as calling the mockRequestResponse() function and then modifying whatever property of the req object needs to be updated.

const { req, res } = mockRequestResponse();
req.query = { gatewayID: 'hello_world' };

To check response objects, especially for error scenarios where different error strings are passed when a gateway ID is missing or invalid, we can use the res._getJSONData() function to actually read out the contents of the response. That way we can check actual error message along with the HTTP status codes.

Pretty handy, right?

Check the test code coverage

If you're using Jest's code coverage reporting features, now's a good time to run that function and check out the code coverage for this file in the terminal printout or the browser.

You can open the code coverage report via the command line by typing: open coverage/lcov-report/index.html

And hopefully, when you navigate to the code coverage for the pages/api/ routes, you'll see some much better code coverage for this file now.

Now go forth and add unit tests to all other API routes as needed.

Conclusion

I'm a fan of the Next.js framework - it's React at its heart with lots of niceties like SEO and API routes baked in. While Next fits the bill for many projects nowadays and helps get us up and running fast with projects, its testing documentation leaves something to be desired - especially for some of its really great additions like API routes.

Automated testing is a requirement in today's modern software world, and being able to write unit tests to continue to confirm an app's functionality works as expected isn't something to be ignored or glossed over. Luckily, the node-mocks-http library helps make setting up mocked req and res objects simple, so that we can test our Next.js app from all angles - from presentational components in the DOM down to API routes on the backend.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading. I hope learning how to unit test API routes helps you out in your next Next.js project (no pun intended!).

References & Further Resources

Create an Asset Tracker with Next.js and React Leaflet

Tue, 18 Jan 2022 00:00:00 GMT

import YouTube from '../../src/components/YouTube.astro';

</figure>

Introduction

Recently I began working for an Internet of Things startup, Blues Wireless that aims to make IoT development easier - even when reliable Internet connections are not available. Blues does this via Notecards - prepaid cellular devices that can be embedded into any IoT device "on the edge" to transmit sensor data as JSON to a secure cloud: Notehub.

Since web development is my main area of expertise (not IoT development), I started off building an easier IoT project: an asset tracker using just a Blues Notecard, Blues Notecarrier AL with a built-in GPS antenna, and a small lithium-ion polymer (LiPo) battery.

With the help of the Blues developer experience documentation, I had GPS location data being delivered to the Notehub cloud in short order. That's cool and all, but the way that data from sensors in the world really becomes useful is when it's displayed to users in some sort of UI, right? It could be charts, tables, or in my case, a map.

So I wanted to take my data from the Notehub cloud and put it into a custom-made dashboard to track and display the Notecard's location in the real world. Since React is my current JavaScript framework of choice, I decided to build a Next.js- TypeScript-powered dashboard, and I learned a ton of stuff in the process, which I intend to share with you over a series of blog posts in the next few months.

In this article, I'm going to show you how to add a map to a Next.js application, pull in location data from a third-party API source, and regularly revalidate the data to update the map when new location data is present.

Here's what the final dashboard looks like - the map is the focus for this particular post:

Set up a map component in Next.js app

NOTE: This article will not go through the initial setup of a brand new Next.js app - that's outside the scope of this post. If you're starting from scratch, I would recommend following the Next.js starter app with TypeScript documentation.

If you'd prefer, you can also fork and download my whole, working code from the GitHub repo here.

Install map project dependencies

The first thing to do in this post, is add a map to a Next project. This is going to require a few new npm packages added to our project: leaflet, react-leaflet and leaflet-defaulticon-compatibility.

$ npm install leaflet react-leaflet leaflet-defaulticon-compatibility

NOTE: You'll also need react and react-dom as peer dependencies if they're not already in your project, too.

leaflet is our base, JavaScript library for interactive maps - it provides the basis upon which our other packages depend.
react-leaflet provides easier-to-use React components for Leaflet maps. It provides bindings between React and Leaflet, not replacing Leaflet, but leveraging it to abstract Leaflet layers as React components. If you're curious to learn more about React Leaflet, I recommend perusing the documentation.
leaflet-defaulticon-compatibility retrieves all Leaflet Default Icon options from CSS, in particular all icon images URL's, to improve compatibility with bundlers and frameworks that modify URL's in CSS.

Build engines and frameworks that modify URLs in CSS, can often conflict with Leaflet built-in Default Icon images automatic management, and this package helps handle it.

TypeScript Note:

If you're using TypeScript in your project, you'll also want to want to install the follow dev dependency to avoid TypeScript errors:

$ npm install @types/leaflet --save-dev

With new map libraries installed, it's time to move on to configuring the project to use these new resources.

Generate a Mapbox token for the map's display style and add it to the project

For the map display that the asset tracker will be on, I chose to use Mapbox styles. It's got a lot of nice map display styles to choose from, and developers can create their own Mapbox API tokens to access these styles by signing up for a free Mapbox account.

After you've signed up and created a new API token, copy the token value - it will be used in the Next.js app. In the Next.js app's next.config.js file at the root of the project, add the API token like so:

next.config.js

/** @type {import('next').NextConfig} */
module.exports = {
  reactStrictMode: true,
  env: {
    MAPBOX_ACCESS_TOKEN:
      "[MAPBOX_TOKEN]",
  },
};

From this file, Next can access the token when it needs to call the Mapbox API endpoint. Next, we'll create the <Map /> component in our project.

Create the `<Map>` component

<figcaption>This is how the component looks rendered out with GPS data.</figcaption>

</figure>

As this is a React project, individual, reusable components are how I like to roll, so inside of the project, create a new file named Map.tsx and paste in the following code. The live code is available by clicking the file title below.

Map.tsx

import {
  MapContainer,
  TileLayer,
  Marker,
  Popup,
  GeoJSON,
  CircleMarker,
} from "react-leaflet";
import "leaflet/dist/leaflet.css";
import "leaflet-defaulticon-compatibility/dist/leaflet-defaulticon-compatibility.css";
import "leaflet-defaulticon-compatibility";

const Map = ({
  coords,
  lastPosition,
  markers,
  latestTimestamp,
}: {
  coords: number[][];
  lastPosition: [number, number];
  markers: [number, number][];
  latestTimestamp: string;
}) => {
  const geoJsonObj: any = [
    {
      type: "LineString",
      coordinates: coords,
    },
  ];

  const mapMarkers = markers.map((latLng, i) => (
    <CircleMarker key={i} center={latLng} fillColor="navy" />
  ));

  return (
    <>
      <h2>Asset Tracker Map</h2>
      <MapContainer
        center={lastPosition}
        zoom={12}
        style={{ height: "100%", width: "100%" }}
      >
        <TileLayer
          url={`https://api.mapbox.com/styles/v1/mapbox/streets-v11/tiles/256/{z}/{x}/{y}@2x?access_token=${process.env.MAPBOX_ACCESS_TOKEN}`}
        />
        <Marker position={lastPosition} draggable={true}>
          <Popup>
            Last recorded position:
            <br />
            {lastPosition[0].toFixed(3)}&#176;,&nbsp;
            {lastPosition[1].toFixed(3)}&#176;
            <br />
            {latestTimestamp}
          </Popup>
          <GeoJSON data={geoJsonObj}></GeoJSON>
          {mapMarkers}
        </Marker>
      </MapContainer>
    </>
  );
};

export default Map;

Now let's talk about everything that's going on in this component.

At the top of the file:

all the individual React Leaflet components needed for this component are imported,
the original Leaflet CSS is imported,
and the Leaflet Default Icon Compatibility CSS and JS are imported afterwards - as specified by the usage instructions.

After that, are the props that this component accepts:

coords - a list of arrays that have GPS longitude and latitude in them - this draws the connecting lines between coordinates.
lastPosition - the most recent GPS latitude and longitude to display in the popup when the user clicks the icon on the map.
markers - another list of arrays that have GPS latitude and longitude (yes, the order of coordinates is reversed in these arrays) to display the blue circles of previous places on the map the tracker was.
latestTimestamp - the most recent timestamp of GPS coordinates received (also for displaying in the popup on the map).

Let's skip down to the JSX.

<MapContainer /> is the component responsible for creating the Leaflet Map instance and providing it to its child components - without this component, the map won't work. In this component we can define the map's center coordinates, its default zoom level on the map, and some basic styling for the component to display properly.

The <TileLayer /> component is where our Mapbox style and newly generated API token come into play. Just choose whatever style suits your fancy, replace the streets-v11 portion of the string with it, and make sure the Mapbox token is present in the next.config.js file, which I showed in the previous step. Without this component there's no map background for the coordinates to render on - instead it will just be a blank canvas.

<Marker /> takes in the lastPosition prop to display the icon on the map of the tracker's last recorded position, and it wraps the <Popup /> component, the <GeoJSON /> component, and the list of <CircleMarker /> components.

The <Popup /> component is a nicely-styled tooltip that can display whatever info is desired. My <Popup /> shows the tracker's last GPS coordinates and time it was reported when a user clicks on it, but it can display anything you want.

The <GeoJson /> component is where the coords list of GPS longitude and latitude arrays are passed in to draw the connecting lines between coordinates. The type: "LineString" in the geoJsonObj where the coordinates are fed in, is what handles it.

And last but not least, the <CircleMarker >/ components, which are displayed in this component's JSX as {mapMarkers}.

NOTE: In order to get all the markers in the list to render as individual circles on the map, I had to create this little function to iterate over the list and generate all the circles, then inject that directly into the JSX.

Trying to iterate over all the values inside the JSX wouldn't work, which I believe is an example of how this react-leaflet package behaves differently from traditional React code.

And that's all that's going on in this component, not so complicated when it's broken down into the individual pieces that make it up, right?

Render the map in a Next.js app

Our final step to get the map rendering inside of a Next app: importing the component with the option ssr:false.

Since the react-leaflet library only works in the browser, we have to use Next.js's dynamic import() support with no SSR to tell the map component to only render after the Next.js server-side rendering has happened.

So wherever this <Map /> component is being injected into your app, use the syntax detailed below. In my app, it's in the index.tsx page file, and I've condensed the code in the file down for clarity. Click on the file title to see the full code.

pages/index.tsx

// imports
import dynamic from "next/dynamic";
// other imports

type dataProps = {
// condensed for code brevity
};

export default function Home({ data }: { data: dataProps[] }) {
  // needed to make the Leaflet map render correctly
  const MapWithNoSSR = dynamic(() => import("../src/components/Map"), {
    ssr: false,
  });

  // logic to transform data into the items needed to pass to the map

return (
    <div>
      {/* extra tracker app code */}
      <main>
        <h1>React Blues Wireless Asset Tracker</h1>
        {/* other tracker components */}
        <div>
          <MapWithNoSSR
            coords={lngLatCoords}
            lastPosition={lastPosition}
            markers={latLngMarkerPositions}
            latestTimestamp={latestTimestamp}
          />
        </div>
        {/* other tracker components */}
      </main>
    </div>
  );
}

// more code down here: getStaticProps

Once the <Map /> has been dynamically imported with server-side rendering disabled, the component can be used just like any other in the application. Simple as that.

At this point, feel free to mock some hard coded data for the map to make sure it's working correctly. In the next section, I'll cover pulling in live data from the Notehub cloud were my asset tracker location data currently lives.

Pull in the data for the map

Ok, so the map is set up in the app, now it's time to give it some data to display. If you'd like to build your own asset tracker like I did, you're welcome to - I'll briefly outline the hardware I used and point you towards the documentation to get it configured - the very same documentation I used to set mine up.

IoT Hardware list

Here's the equipment you'll need to make this project happen:

Blues Wireless NB-IoT & LTE-M Notecard Global
Blues Wireless Notecarrier AL with LiPo battery connector
LiPo battery from an IoT supplier like Adafruit (mine is the 3.7v 2500mAh version)

Initial Notecard and Notecarrier AL configuration

Follow the Blues quickstart guide to set up the Notecard, Notecarrier, and first Notehub project, and the Blues asset tracking guide for the commands needed to configure a Notecard for GPS location tracking.

In addition to these instructions, there's an important caveat to make this asset tracker work for our purposes.

In the final configuration step: card.location.track where the tracker starts running, include the property of: "sync" : true. This property means as soon as a new event is acquired by the Notecard (a new GPS location, in this case), the Notecard will sync the event to Notehub instead of waiting for its regularly scheduled outbound time.

If you're curious, here's all of the commands I used to set up my Notecard from start to finish using the built-in web REPL on the Blues developer experience site.

$ {"req":"card.restore","delete":true}
#factory reset Notecard

$ {"req":"hub.set","product":"com.blues.[NOTEHUB_PROJECT_ID_HERE]","mode":"periodic","outbound":10,"inbound":60}
#attach tracker to Notehub project, set it to periodic mode,
#sync outbound requests from the Notecard every 10 mins and inbound reqs from Notehub every 60 mins

$ {"req":"card.location.mode","mode":"periodic","seconds":360}
#tell card how often to get GPS reading and only when motion is detected

$ {"req":"card.location.track","start":true,"heartbeat":true,"hours":12,"sync":true}
#start tracking, issue heartbeat every 12 hours when no motion detected,
#sync data with Notehub as soon as a tracking event is acquired (this is an important one)

Use the Notehub API to fetch the tracker data into Next.js

The Notehub API can be used to fetch events (the data containing Notecard GPS coordinates) directly from Notehub. The Notehub API requires users to create an authorization token to be passed along with requests, but the process is well documented, as is the API to fetch all events.

Generate a Notehub auth token

Below is the code to run in the command line to generate the Notehub authorization token:

$ curl -X POST
-L 'https://api.notefile.net/auth/login'
-d '{"username":"[you@youremail.com]", "password": "[your_password]"}'

Copy this token and inside of your Next.js project, at the root of the project, create a .env.local file - this is where sensitive info will be kept: secrets, project info, and anything else you'd rather not commit to GitHub for all the world to see. Here's an example of what the file should look like, and the two secret variables it needs:

.env.local

NOTEHUB_PROJECT_ID=APP_ID_GOES_HERE # get this from Notehub
NOTEHUB_TOKEN=NOTEHUB_GENERATED_TOKEN_GOES_HERE # paste in token generated in previous step

This .env.local file is how Next.js automatically reads in environment variables used at build time or on the client side. All the variables you'll need are build time variables so none of them need to be prefixed with NEXT_PUBLIC_, which allows for variable access on the client side.

Create a fetchNotecardData() function in Next.js

With our Notehub token and project ID specified in the .env.local file, now we can make our connection to Notehub via Next's getStaticProps function.

Create a new file named something like notecardData.ts, this is where the function to fetch data from Notehub and filter down to the events we want - the _track.qo events - will live.

Here is what the code to fetch events will look like - click on the file name to see the code in my actual repo.

notecardData.ts

export async function fetchNotecardData() {
  interface dataProps {
    [file: string]: any;
  }

  let eventArray: object[] = [];
  const baseUrl = `https://api.notefile.net/v1/projects/${process.env.NOTEHUB_PROJECT_ID}/events`;

  const headers = {
    "Content-Type": "application/json",
    "X-SESSION-TOKEN": `${process.env.NOTEHUB_TOKEN}`,
  };

  const res = await fetch(fullUrl, {
    headers: headers,
  });
  const eventData = await res.json();
  eventArray = eventData.events;

  while (eventData.has_more) {
    const res = await fetch(`${baseUrl}?since=${eventData.through}`, {
      headers: headers,
    });
    const newEventData = await res.json();
    eventArray = [...eventArray, ...newEventData.events];
    if (newEventData.has_more) {
      eventData.through = newEventData.through;
    } else {
      eventData.has_more = false;
    }
  }

  const filteredEvents = eventArray.filter(
    (event: dataProps) => event.file === "_track.qo"
  );

  return filteredEvents;
}

Although this file looks verbose at first glance, it's not actually that complex.

It starts off setting up a baseUrl that defines the URL connection to Notehub events - this is where access the NOTEHUB_PROJECT_ID environment variable comes into play.

Then the header object containing the "X-SESSION-TOKEN", which is set equal to our generated NOTEHUB_TOKEN, is up next.

After that, the Notehub event endpoint is hit, and it will automatically pull back the first 50 events it has stored, and if there's more events than what was just returned, the JSON response list will also include the properties through and has_more.

If has_more exists, this while loop function will keep hitting Notehub using the through value (the globally-unique identifier of the last event in the array of returned events), until there are no more events to add to the eventArray list.

Finally, all of the events that have been gathered up are filtered down to only the _track.qo type, because those are the events that contain the tracker's GPS coordinates.

Call the `fetchNotecardData()` function on the page

With our function to connect to Notehub and get event data constructed, it's time to make that call in the page where the <Map /> component is located. To do this, we'll be using Next's getStaticProps function to fetch the data server-side.

Inside of the index.tsx file where we previously imported the <Map /> component, we'll add the following code down at the bottom of the file. I've condensed the rest of the file for clarity, but the full file is linked below.

pages/index.tsx

// imports
import { GetStaticProps } from "next";
import { fetchNotecardData } from "../src/lib/notecardData";
// other imports

type dataProps = {
// condensed for code brevity
};

export default function Home({ data }: { data: dataProps[] }) {
  // map component imported dynamically here

  // logic to transform data into the items needed to pass to the map

return (
    <div>
      {/* extra tracker app code */}
      <main>
        <h1>React Blues Wireless Asset Tracker</h1>
        {/* other tracker components */}
        <div>
          <MapWithNoSSR
            coords={lngLatCoords}
            lastPosition={lastPosition}
            markers={latLngMarkerPositions}
            latestTimestamp={latestTimestamp}
          />
        </div>
        {/* other tracker components */}
      </main>
    </div>
  );
}

export const getStaticProps: GetStaticProps = async () => {
  const data = await fetchNotecardData();
  return { props: { data } };
};

To call the Notehub API in the index.tsx file on the server-side, we import the fetchNotecardData() function itself, import the getStaticProps function from Next, and then at the end of the file, call fetchNotecardData() from inside the getStaticProps function.

Finally we return that data from Notehub as props that can be passed to the Home component.

Massage the data into shape for the `<Map />` component

Good. We've got data from Notehub, and there's one last thing to do: take this JSON data returned from Notehub and re-shape it to fit the <Map /> component. Once again, I've condensed down the logic to make this file easier to read, but you can see the full file on GitHub.

pages/index.tsx

// imports
import { useEffect, useState } from "react";
import dayjs from "dayjs";
// other imports

type dataProps = {
// condensed for code brevity
};

export default function Home({ data }: { data: dataProps[] }) {
  // map component imported dynamically here

  // state variables for the various pieces of data passed to the map
  const [lngLatCoords, setLngLatCoords] = useState<number[][]>([]);
  const [lastPosition, setLastPosition] = useState<[number, number]>([
      00.00, 00.00
  ]);
  const [latestTimestamp, setLatestTimestamp] = useState<string>("");
  const [latLngMarkerPositions, setLatLngMarkerPositions] = useState<
    [number, number][]
  >([]);

  // logic to transform data into the items needed to pass to the map
  useEffect(() => {
    const lngLatArray: number[][] = [];
    const latLngArray: [number, number][] = [];
   
    if (data && data.length > 0) {
      data
        .sort((a, b) => {
          return Number(a.captured) - Number(b.captured);
        })
        .map((event) => {
          let lngLatCoords: number[] = [];
          let latLngCoords: [number, number] = [0, 1];

          lngLatCoords = [
            event.gps_location?.longitude,
            event.gps_location?.latitude,
          ];
          latLngCoords = [
            event.gps_location?.latitude,
            event.gps_location?.longitude,
          ];
          
          lngLatArray.push(lngLatCoords);
          latLngArray.push(latLngCoords);
        });
      const lastEvent = data.at(-1);
      let lastCoords: [number, number] = [0, 1];
      lastCoords = [
        lastEvent.gps_location.latitude,
        lastEvent.gps_location.longitude,
      ];
      setLastPosition(lastCoords);
      const timestamp = dayjs(lastEvent?.captured).format("MMM D, YYYY h:mm A");
      setLatestTimestamp(timestamp);
    }
    setLngLatCoords(lngLatArray);
    setLatLngMarkerPositions(latLngArray);
  }, [data]);

return (
    <div>
      {/* extra tracker app code */}
      <main>
        <h1>React Blues Wireless Asset Tracker</h1>
        {/* other tracker components */}
        <div>
          <MapWithNoSSR
            coords={lngLatCoords}
            lastPosition={lastPosition}
            markers={latLngMarkerPositions}
            latestTimestamp={latestTimestamp}
          />
        </div>
        {/* other tracker components */}
      </main>
    </div>
  );
}

// getStaticProps call to Notehub

In this function, once the data is fetched from Notehub and passed to the component, we set some new React useState variables to hold the data to pass to the <Map /> component.

There's a lngLatCoords list - this is list of coordinates that will be used to draw the lines between the recorded GPS coordinates (and it must be passed as [longitude, latitude] to the "LineString" <GeoJSON /> component).
The lastPosition state variable is used to center the map, center the marker icon, and display in the <Popup /> along with the latestTimestamp variable.
And the latLngMarkerPositions list is similar to the lngLatCoords variable except the coordinates here are in the order of [latitude, longitude].

Inside of the useEffect() function the array of Notehub events is sorted and then iterated over to pull out all the relevant data in the shape required. And once all the events have been transformed, they're set in state and passed to the map.

Update the data regularly using ISR

There's one last thing we have not yet discussed and that is how to handle new _track.qo events that get sent to Notehub after the asset tracking app originally loads the data and renders the map.

Early on as I was building my tracker, I had a simple refreshData() function I was using to force Next.js to refetch Notehub data on the server-side on an interval every 5 minutes, but then I learned a much better way that is actually built in to Next: incremental static regeneration (ISR).

ISR enables us to use static-generation on a per-page basis, without needing to rebuild the entire site. What this means in practice is: getStaticProps is still called to fetch the data, but additionally a revalidate option is passed in to the return statement with an interval time in seconds. Every time that interval is reached, Next.js will call the function again and attempt to regenerate the page with any new data, and once the page has been regenerated Next will replace the old page with the new one. No loading messages, no screen jank, no extra libraries like WebSockets or long-polling or extra functions to force server-side refreshes manually - instead it's built in to Next.

So to enable this revalidation of the page data on a regular interval, we'll turn back to the index.tsx page once more.

pages/index.tsx

// imports
import { GetStaticProps } from "next";
import { fetchNotecardData } from "../src/lib/notecardData";
// other imports

type dataProps = {
// condensed for code brevity
};

export default function Home({ data }: { data: dataProps[] }) {
  // map component imported dynamically here

  // logic to transform data into the items needed to pass to the map

return (
    <div>
      {/* extra tracker app code */}
      <main>
        <h1>React Blues Wireless Asset Tracker</h1>
        {/* other tracker components */}
        <div>
          <MapWithNoSSR
            coords={lngLatCoords}
            lastPosition={lastPosition}
            markers={latLngMarkerPositions}
            latestTimestamp={latestTimestamp}
          />
        </div>
        {/* other tracker components */}
      </main>
    </div>
  );
}

export const getStaticProps: GetStaticProps = async () => {
  /* we're able to use Nextjs's ISR (incremental static regneration) 
  revalidate functionality to re-fetch updated map coords and re-render one a regular interval */
  const data = await fetchNotecardData();

  return { props: { data }, revalidate: 120 };

The new code is at the very bottom of the getStaticProps function in this file - the line return { props: { data }, revalidate: 120 };

This is all that's needed so that every two minutes Next.js will go back to Notehub and fetch any new data and re-render the page server-side. It's awesome.

And with that, we've got an asset tracker map built with Next.js, and regularly checking for new data.

Conclusion

After I joined an IoT startup in July of 2021, I started dipping my toes into internet of things development with an asset tracker. And once I had GPS data being sent regularly to a cloud, I figured out how to extract that data from the cloud and display it in a custom-built dashboard map.

With the help of the React-powered Next.js framework and the React Leaflet library, I was able to do so easily, and even leverage Next's built-in incremental static regeneration to re-fetch any new data server-side and re-render the map with that fresh data. It's pretty cool.

This actually came in pretty handy when my parents' car was stolen from their driveway the night after Thanksgiving. If you want to hear the whole story and build your own tracker, check out this blog post and video I made for Blues Wireless - it details the whole process from hardware to software to deploying to Netlify.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading. I hope you enjoyed learning how to set up a map in Next.js and render an asset tracker's location data to that map - just think how useful this could be for keeping track of a personal vehicle or a whole fleet of them. There's a lot of cool places you could take this project from here. Happy tracking!

References & Further Resources

Hackster.io original asset tracker project
Asset tracker GitHub repo
Leaflet documentation
React Leaflet documentation
Mapbox site
Blues Wireless website
Blues Wireless developer experience site
Next.js incremental static regeneration

Build a Gatsby Blog Post Filter

Thu, 16 Dec 2021 00:00:00 GMT

import YouTube from '../../src/components/YouTube.astro';

</figure>

Introduction

Earlier this year, I built my own website using Gatsby to serve as a centralized place to showcase my work, my talks, my blog articles and courses, and let people get to know me a little better.

Since I've been writing regularly for almost 4 years now, there's a lot of blog articles to sort through on the site, so one thing I wanted to do was make a filter on the blog page to let readers narrow down what they were interested in. There's a catch to this filter though, and it's twofold:

I wanted to give readers multiple parts of a blog they could search by: blog titles, blog subtitles, and blog tags.
Not all of my blog posts live on my site: many are hosted on other platforms like Medium, Hackster.io, etc. so I couldn't rely solely on Gatsby's GraphQL queries during page generation to grab all the post data needed.

It makes for a tricky situation, but certainly not an insurmountable one.

In this post, I'll show you how to build a blog filter in Gatsby that accesses data from multiple sources and filters them by multiple parameters.

Here's what the finished product looks like:

First things first, get all the post data in one place

Before we can start filtering down the blog posts we must gather together all the data from their different sources.

If we were using just Gatsby to host blog content in the form of local Markdown files or a headless CMS integration, it would be possible to use Gatsby's gatsby-source-filesystem or the Gatsby CMS integration guide of your choice to pull in all the data in one fell swoop, but what about blogs that aren't accessible this way?

Before I built this site, I posted almost all of my tech blogs on Medium. I'm in the process of moving them all here, but it's a lot of content and the going is slow.

Well here's my low-tech, interim solution: a simple JavaScript list.

Make a file for the blogs not hosted on our Gatsby site

Until I eventually get all of my blogs hosted in one place (this website), I came up with an easy, workable solution - there's not even any GraphQL queries required. My solution was creating a new JavaScript file named mediumBlogs.js. And all the file is, is a list of all my blogs on third-party sites.

Here's an example of a few of the posts and the properties I include for each blog post.

mediumBlogs.js

import moment from 'moment';
import googleJib from '../content/images/google-jib.png';
import sequelize from '../content/images/sequelize.png';

export default [
  {
    date: moment('2018-07-18').format('ll'),
    img: googleJib,
    url:
      'https://itnext.io/jib-getting-expert-docker-results-without-any-knowledge-of-docker-ef5cba294e05',
    subTitle: 'All of the containerization benefits, none of the complexity.',
    tags: ['docker', 'devops', 'java ', 'jib'],
    timeToRead: 4,
    title: 'Jib: Getting Expert Docker Results Without Any Knowledge of Docker',
  },
  {
    date: moment('2018-08-04').format('ll'),
    img: sequelize,
    url:
      'https://medium.com/@paigen11/sequelize-the-orm-for-sql-databases-with-nodejs-daa7c6d5aca3',
    subTitle: 'The ORM For SQL Databases with Node.js',
    tags: ['javascript', 'nodejs', 'sql', 'express'],
    timeToRead: 8,
    title: 'Sequelize: Like Mongoose But For SQL',
  },
  // more blogs following the same format here
]

Unfortunately there was no Medium API to pull in this sort of data programmatically, so I made up my own properties of info I wanted to include - things like date, original URL, title and subtitle, tags, a thumbnail image - you get the idea, and manually put the file together.

I may have been able to figure out a more automated way to scrape this data from Medium, but it probably would have taken me just as long to come up with a working script as it did to type up this file with links to all my posts. I wasn't in a massive rush so I would just add 5 posts at a time as I was working on building the site in my spare time.

The nice thing about doing it this way is that I was able to create my custom list of blog posts with exactly the data I wanted in the shape I needed - a shape that can be closely mimicked in my GraphQL queries to fetch locally hosted blog data, which I'll cover in the next sections.

To use this file anywhere in the Gatsby blog, it just needs to be imported into the component that needs it.

My list of blogs live in the PostListing component, so to get the Medium blog list it's as simple as importing anything else into a JS file:

PostListing.js

// other file imports above here
import mediumBlogs from '../../../data/mediumBlogs';
// other file imports below here

With the third-party post data handled, it's time to move on to the local data in Markdown files on the site.

Create a GraphQL `useStaticQuery` Hook to grab all the local data

After making a list of all the additional blog posts not accessible by the Gatsby site, we also need to grab the data that Gatsby is responsible for. Since I choose to host my blogs in Markdown files that are stored in a private repo on GitHub, this data can be accessed with a combination of the gatsby-source-filesystem in the gatsby-config.js file, and a custom Hook on the frontend to query the data.

Using the gatsby-source-filesystem to generate content from source data like Markdown files is fairly straightforward following the documentation on the Gatsby site: just point the filesystem plugin towards your directory folder where all the files live and it takes care of the rest.

On the client side, my version of Gatsby (v2.1 and above) supports GraphQL useStaticQuery Hooks that can run at build time. The special thing about these queries is that they don't need to be at the page level and they can be written as reusable custom Hooks not attached to a particular page or component.

There are a few limitations to useStaticQuery that makes it slightly less flexible than original, page-level queries, but it will work just fine for our purposes today.

If you'd like to know more about this Hook, I wrote about it in-depth here.

Since the front matter in my local Markdown post files looks like this:

---
title: "Docker 102: Docker-Compose"
subTitle: 'The recipe card for getting all your Dockerized apps to work together seamlessly.'
thumbnail: ../images/docker-2.png 
featuredImage: ../images/docker-2.png
category: "devops"
date: "2018-07-07"
ogLink: 'https://itnext.io/docker-102-docker-compose-6bec46f18a0e'
publication: ITNEXT
tags:
    - docker
    - devops
omit: false
---

The custom useStaticQuery Hook to fetch and parse this data can look something like this:

usePostListingQuery.js

import { useStaticQuery, graphql } from 'gatsby';

export const usePostListingQuery = () => {
  const postListData = useStaticQuery(graphql`
    query PostQuery {
      allMarkdownRemark(
        filter: { frontmatter: { omit: { eq: false } } }
        sort: { fields: [fields___date], order: DESC }
      ) {
        edges {
          node {
            fields {
              slug
              date(formatString: "MMM D, YYYY")
            }
            timeToRead
            frontmatter {
              omit
              title
              subTitle
              tags
              date
              thumbnail {
                childImageSharp {
                  fixed(width: 200) {
                    ...GatsbyImageSharpFixed
                  }
                }
              }
            }
          }
        }
      }
    }
  `);
  return postListData;
};

This query fetches all the same data from the Markdown files' front matter as the data properties included in the custom list of third-party blog posts we made.

With this reusable custom Hook at our disposal, it can be imported into the PostListing component to fetch all the data about our local blog posts.

PostListing.js

// other imports above
import { usePostListingQuery } from '../../Hooks/usePostListingQuery';
// other imports below

const PostListing = () => {
  const localSitePosts = usePostListingQuery();

Combine the data into one list

Now that the PostListing component's getting data from both our data sources, it's time to put them together in the right order. Still in the PostListing component, we can declare a useState Hook called fullPostList, which is what will be rendered in the component's JSX. And inside of a useEffect Hook, which will run on component load, we'll call a function named getAndFormatAllPosts() which will combine the localSitePosts queried via GraphQL from the Markdown files and the mediumBlogs list from the mediumBlogs.js file.

PostListing.js

  const [fullPostList, setFullPostList] = useState([]);

  // get all Markdown post date with this custom Hook
  const localSitePosts = usePostListingQuery();

  const getAndFormatAllPosts = (posts) => {
    const postList = posts.allMarkdownRemark.edges.map((postEdge) => {
      return {
        path: postEdge.node.fields.slug,
        tags: postEdge.node.frontmatter.tags,
        thumbnail: postEdge.node.frontmatter.thumbnail.childImageSharp.fixed,
        title: postEdge.node.frontmatter.title,
        subTitle: postEdge.node.frontmatter.subTitle,
        date: postEdge.node.fields.date,
        timeToRead: postEdge.node.timeToRead,
      };
    });
    // access the Medium blog list imported into the component
    const fullPostList = postList.concat(mediumBlogs);
    // sort the combined posts by date
    const sortedPostsList = sortArrayByDate(fullPostList); 
    // set the full array of posts into component state
    setFullPostList(sortedPostsList);
  };

  useEffect(() => {
    // combine and format all the blog posts on component load
    getAndFormatAllPosts(localSitePosts);
  }, []);

First, getAndFormatAllPosts() loops through all the Markdown posts fetched by the usePostListingQuery() Hook and creates a list of objects with properties to match the mediumBlogs' object properties.

Then, once all the localSitePosts are formatted, the mediumBlogs list is combined with them, and everything is sorted by date (that's the sortArrayByDate() helper function), and this full list is set in our local state Hook fullPostList.

Whew! Quite a lot happened here, but now we have a single list of objects with the same properties, time to filter them down.

Set up an input filter above the blog posts

Ok, there's our whole list of blogs, sorted and rendered in our PostListing component's JSX. The next step is adding an input at the top of the component that a user can type into.

If you're familiar with React and inputs, you know that the only way to register keystrokes in an input is with an onChange() function that takes in events from the DOM, and then updates the component's state based on those events. For our purposes, we need the component to use whatever's in the input and narrow down the blog posts visible based on their title, subTitle and tag properties.

Add variables for the query values

To help us know when to show the full list of posts versus a narrowed down list based on if the input is empty or not, we need to add a couple of new variables to our component. One is a constant named emptyQuery to set the input's value to an empty string when the component first loads, and the second is a new useState Hook that starts out as an object with an empty filteredPostList array and a query property that defaults to the value of emptyQuery.

PostListing.js

  const emptyQuery = '';
  const [state, setState] = useState({
    filteredPostList: [],
    query: emptyQuery
  });

Create a `filterPosts()` function

Our new variables enable us to write an onChange() function for the input that will let us narrow down the results and return them to the component to display.

We're going to name this function filterPosts(), and every time a new keystroke (event) is added to the input this function will run. It will start by setting the current event.target.value from the DOM equal to a variable locally scoped to the function named query.

Next, we'll take the fullPostList state and filter through all the posts looking to see which post.title, post.subTitles and post.tags match the query. Don't forget to use something like toLowerCase() to ensure case sensitivity doesn't accidentally disqualify a post from making the list.

Once the local query variable and new filteredPostList array have been created in the filterPosts() function, the component's state object is set with the new values.

PostListing.js

  const filterPosts = (event) => {
    const query = event.target.value;
    const filteredPostList = fullPostList.filter((post) => {
      return (
        post.title.toLowerCase().includes(query.toLowerCase()) ||
        (post.subTitle &&
          post.subTitle.toLowerCase().includes(query.toLowerCase())) ||
        (post.tags &&
          post.tags.join('').toLowerCase().includes(query.toLowerCase()))
      );
    });

    // set the component's state with our newly generated query and list variables
    setState({
      query,
      filteredPostList
    });
  };

This function's not so complicated, right? In reality we just have to check multiple properties in each post object in the array to see if our query exists in any of them. The post.tags is a little tricky because it's an array of strings instead of a single string, but simply using join('') on the tags before checking to see if the query is included takes care of this too.

Render the newly filtered list

The final step is to take our new filteredPostList and query pieces of state, determine if the query is not an empty array (which would mean the search input is empty), and based on that render either the fullPostList or filteredPostList in the component.

To do this part, we can create a new boolean called hasSearchResults that will simply indicate if the filteredPostList exists and the query is not an empty string, and based on this boolean, we'll use another local variable called posts that will tell the component which list to use.

This is how we'll use just one local variable, posts, to render our list of blog posts, regardless of which state variable is actually being rendered by the JSX (fullPostList or filteredPostList). Check out this code snippet to see what I mean.

PostListing.js

  const { filteredPostList, query } = state;
  const hasSearchResults = filteredPostList && query !== emptyQuery;
  const posts = hasSearchResults ? filteredPostList : fullPostList;

 return (
    <>
      {!posts.length && query === emptyQuery && (
        <span className="post-wrapper">
        <span className="post-search-wrapper page-body">
            <input
              className="searchInput"
              type="search"
              aria-label="Search"
              placeholder="Filter blog posts by title or tag"
              onChange={(e) => filterPosts(e)}
            ></input>
          </span>
          <div className="posts-wrapper wide-page-body">
            {posts.length ? (
              posts.map((post, index) => (
                {/* more JSX code to render posts */}

Pretty cool, huh?

Handle no results

There's one last scenario left to handle: if the search term in the input doesn't match any of the values in the blog posts. Luckily, that's fairly straightforward to deal with as well.

For this situation, we can let the JSX do the heavy lifting - no extra state variables required. When the possibility occurs that whatever input string a user has searched for doesn't exist in any of the posts being filtered through, we need to show no posts and let the user know nothing matches their search params.

The simplest way to do this in the JSX is by checking the posts.length, which we already do in order to map over all the blog posts and display them on the page. We'll take this statement's ternary and change it from rendering null if posts.length does not exist and make it render a message for the user instead.

Here's what the whole component's JSX looks like (slightly condensed for brevity): as long as the blog posts have loaded but the amount of ones filtered according to the user input is 0, the message displayed to the user says "Sorry, no search results match your query.".

PostListing.js

return (
  <>
    {!posts.length && query === emptyQuery && (
      <span className="post-wrapper">
        <span className="post-search-wrapper page-body">
          <input
            className="searchInput"
            type="search"
            aria-label="Search"
            placeholder="Filter blog posts by title or tag"
            onChange={(e) => filterPosts(e)}
          ></input>
        </span>
        <div className="posts-wrapper wide-page-body">
          {posts.length ? (
            posts.map((post, index) => (
              <article className="post" key={index}>
                <p className="post-date">
                  {post.date}&nbsp;
                  {'\u2022'} 
                  {post.timeToRead} min read
                </p>
                <Link to={`/blog${post.path}`} key={post.title}>
                  <p className="post-title">{post.title}</p>
                  <Img fixed={post.thumbnail} />
                  <p>{post.subTitle}</p>
                </Link>
                <PostTags tags={post.tags} />
              </article>
            ))
          ) : (
            <div className="empty-results">
              <h2>Sorry, no search results match your query.</h2>
            </div>
          )}
        </div>
      </span>
    </>
  )}
);

Isn't it nice how cleanly that can be handled with a ternary? I like it when React makes things easy for me.

Conclusion

Filtering and inputs are not the simplest thing to build when it comes to web development, and filters that check multiple properties in multiple objects for a match are even harder. But it can be done.

Giving users the ability to narrow down results based on their interests (which hopefully align with article titles, subtitles, and tags) is a really nice feature to offer, and it was a fun challenge to build in Gatsby. And, surprisingly, not as complicated as I originally expected it to be.

Check back in a few weeks — I’ll be writing more about JavaScript, React, ES6, or something else related to web development.

Thanks for reading. I hope you can put this custom built filter to use in your own React applications. I know I'm always grateful for filter inputs like this and I think your users will be too.

References & Further Resources

GitHub repo for my personal site

Cypress Studio - the underrated feature speeding up e2e testing

Thu, 21 Oct 2021 00:00:00 GMT

<figcaption>Photo by <a href="https://unsplash.com/@euwars">Farzad Nazifi</a> on Unsplash</figcaption>

</figure>

Cypress.io just keeps raising the bar on end-to-end testing

When Cypress.io first hit the scene in 2015, it made a splash because it fixed so many of the issues that existed with other end-to-end testing (e2e) competitor frameworks.

Between good documentation, intuitive syntax, improved debugging, and no reliance on Selenium under-the-hood - everything about Cypress was a major step forward for e2es, but it wasn't content just to stop there.

The team behind Cypress regularly keeps releasing new features and functionality to make it more and more useful for devs, and make e2e testing (traditionally kind of a pain) easier and easier as well.

One recent release that's currently tucked behind a feature flag is called Cypress Studio, and it's an absolute game changer.

With Cypress Studio, you can show Cypress exactly how to test by recording interactions against the application under test. You click around in the DOM, and Cypress generates the test code to mimic what you're doing. Test written. Done.

Today, I'll show you how to add Cypress to an existing JavaScript project, enable Cypress Studio, and let it help do the heavy lifting of writing end-to-end tests.

It's such a cool feature to help dev teams save time on testing, deliver new features faster, and still ensure the mission critical functionality of the app continues to work as expected.

Add Cypress to a Project

Although Cypress is kind enough to provide a host of sample scripts to show many of its features in action, it really shines with a local app to test against, and I just so happen to have one that fits the bill.

If you'd like to download a working version of the finished code and follow along, I have a link to the GitHub repo here.

Get an app to test

The app I'm using is a React-based movie database that allows users to see upcoming movies and movies in theaters now, browse movies by genre, and search for movies by title. This will be a good app to demonstrate Cypress Studio's power.

Download Cypress to your app

Once we've got an app to add Cypress to, the first thing we'll need to do is download Cypress to it.

This is another reason Cypress stands head and shoulders above its competitors: one npm download gives you all the tools you need to start writing e2es. No dev dependencies, no extra libraries with mismatched package versions, none of that nonsense to deal with.

At the root of your project, where your package.json file lives, run the following command from the terminal:

$ npm install cypress

This will add a bunch of new Cypress-based folders and files to your app, and with just a few small configuration changes we'll be ready to go.

See all those new folders under cypress/? That's what you should see after initial installation.

Add Cypress scripts to the `package.json` to make it easier to run

For ease of use, I like to add npm scripts for the two main Cypress commands we'll be using:

cypress open, which is used to open the Cypress Test Runner where we can watch tests run locally, debug them, and where we'll show Cypress how to test our app
cypress run, which runs our e2es in "headless mode", which is how they'd run in a build pipeline

In your package.json file, add the following two lines in your "scripts" section.

    "cy:run": "cypress run",
    "cy:open": "cypress open"

Now when we need to run the tests, a simple npm run cy:run or npm run cy:open, straight from the command line, will do the trick.

Run Cypress's test scripts to make sure everything works

Ok, before we get to writing our own tests, let's run the pre-populated tests in Cypress to get familiar with the its Test Runner.

From your command line, run the following shell command:

$ npm run cy:open

This should open the Cypress Test Runner, and from here, click the Run integration spec button in the top right hand corner to run through all the pre-made tests once.

NOTE: Run cypress open first

When we run cypress open for the first time, a new cypress.json file is created at the root of our project. Even though it begins its life empty, this file is required for the headless version of Cypress (cypress run) to work.

So don't try to run Cypress in headless mode until after you've done cypress open in your project at least once. Otherwise, it will throw an error in the console.

After all the Cypress tests run and pass, we're ready to delete them and get to work on our own tests for our app.

Enable Cypress's experimental feature mode

Go ahead and clear all the files out of the Cypress folders of fixtures/ and integration/.

You'll probably also want to add the folders of cypress/screenshots/ and cypress/videos/ to your .gitignore file just so you don't commit those screenshots and videos that Cypress automatically takes during test runs to your GitHub repo (unless you want to, of course).

Add baseUrl variable

With that taken care of, let's set up a baseUrl in our cypress.json file and enable Cypress Studio there too.

Setting this baseUrl variable just means we won't have to type cy.visit('http://localhost:3000/'); in every test. Instead, we'll be able to just write cy.visit('/'); and save ourselves some code.

Turn on experimentalStudio

To enable Cypress studio, just add "experimentalStudio": true to our cypress.json file.

So here's what the cypress.json file will end up with.

{
 "baseUrl":"http://localhost:3000", 
 "experimentalStudio": true
}

Now we can write our first test file and test.

Create a new test file

Inside of the cypress/integration/ folder in your project, create a new test file named movie-search-spec.js. This folder is where Cypress will look for all your e2e test files when it runs.

Give it a placeholder test: we have to tell Cypress where we want it to record the test steps we're going to show it.

So just create a typical describe test block and inside of that, create an it test.

Cypress accepts all the traditional testing syntaxes familiar to those who've used Jasmine, Chai, Mocha and Testing Library testing frameworks before. Whether you prefer describe, it, or test - they're all good.

A good first test would be to test that a user can navigate to the movie search option, search for a particular movie name, and click into the results based on that search.

Here's what my empty testing placeholder looks like in the movie-search-spec.js file.

describe('Movie app', () => {
  it('should show all results for movies that contain "Star Wars" in their titles', () => {
    
  })
})

I think we're about ready to go.

Start the app & show Cypress how to test

One thing you must do before starting up Cypress to run tests against your local app is to start the app locally.

For Cypress, it's an anti-pattern to start the app from a test, so just fire it up in a separate terminal, then open up the Cypress Test Runner.

Start the movie app

In one terminal run our movie app:

$ npm run start

Start the Cypress Test Runner

And in a second terminal, open the Cypress Test Runner:

$ npm run cy:open

In Cypress, Add Commands to Test

When the Cypress Test Runner is open, enter our test file and click the tiny blue magic wand that says Add Commands to Test when you hover over it.

And from here, go for it - test the app.

Go to town

For me, I clicked the Movie Search link in the nav bar, typed "Star Wars" into the search box, clicked into one of the results, etc.

When you're satisfied with what your test is doing, click the Save Commands button at the bottom of the test, and Cypress will run back through all the commands it's just recorded from your actions.

Tell me that's not cool.

Check out the test in your e2e file now

If you go back to your IDE now, you'll see all the actions Cypress recorded, along with a few comments to tell you it was Cypress generating the code and not a developer.

This is what my test now looks like:

describe('Movie app', () => {
  it('should show all results for movies that contain "Star Wars" in their titles', () => {
    /* ==== Generated with Cypress Studio ==== */
    cy.visit('/');
    cy.get('.navbar-top-links > :nth-child(3) > a').click();
    cy.get('.search-input').clear();
    cy.get('.search-input').type('Star Wars');
    cy.get('[type="submit"]').click();
    cy.get(':nth-child(2) > .breakpoint__medium-up > .movie-component > .movie-poster').click();
    cy.get('.movie-details-reviews > :nth-child(3)').click();
    cy.get('.fa').click();
    /* ==== End Cypress Studio ==== */
  })
})

Just wow, right?

Fill in the gaps Cypress didn't code for

Although our test is good, Cypress can't be expected to test for all the things a developer might know are important.

Things like the number of movies returned from searching "star wars" or checking the title of the movie being clicked into and the contents inside of the movie page itself.

I'll fill in some of those details myself now.

describe('Movie app', () => {
  it('should show all results for movies that contain "Star Wars" in their titles', () => {
    /* ==== Generated with Cypress Studio ==== */
    cy.visit('/');
    cy.get('.navbar-top-links > :nth-child(3) > a').click();
    cy.get('.search-label').should('contain', 'Search Movie Titles Here:'); // user added
    cy.get('.search-input').clear();
    cy.get('.search-input').type('Star Wars');
    cy.get('[type="submit"]').click();
    cy.get('.card-component').should('have.length', 20); // user added
    cy.get(':nth-child(2) > .breakpoint__medium-up > .movie-component > .movie-poster').click();
    cy.get('.movie-details-info__overview').should('contain', 'Movie Overview'); // user added
    cy.get('div').should('contain', "1977-05-25") // user added
    cy.get('div').should('contain', "8.2") // user added
    cy.get('.movie-details-reviews > :nth-child(3)').click();
    cy.get('.fa').click();
    /* ==== End Cypress Studio ==== */
  })
})

If you look at my code above, I added comments after the extra assertions I added - mainly small things like checking for search text, the count of movies returned, the movie info like rating and release date in this specific movie.

I didn't add a ton of extra code, just a few extra lines of details. Now run the test again and check the results.

$ npm run cy:run

And we're done! Congrats - our first Cypress Studio-assisted e2e test is written.

Rinse, Repeat

Just repeat these steps for as many end-to-end tests as you need to write and prepare to be amazed at how much time it saves you.

Conclusion

In modern software, testing is a critical piece of any solid enterprise application. It helps ensure our app continues to function as expected while adding new functionality, and end-to-end testing is the closest we can get to mimicking exactly how a user would use the app with automation.

Cypress broke the mold of what e2e testing frameworks are capable of when it debuted back in 2015, and it's only continued to improve with time.

My favorite new feature of late is the ability to show Cypress how a test should act instead of writing it yourself with Cypress Studio - the time saving possibilities are immense. And more time saved means finishing features faster and getting new functionality into the hands of users quicker. Win win win.

If you enjoyed this, you may be interested in checking out my course "The newline Guide to Modernizing an Enterprise React App".

In 10 modules and 54 lessons, I cover all the things I learned while at The Home Depot, that go into building and maintaining large, mission-critical React applications - because it's so much more than just making the code work.

References & Further Resources

Introducing Volta - it manages your Node.js versions so you don't have to

Mon, 18 Oct 2021 00:00:00 GMT

<figcaption>Photo by <a href="https://unsplash.com/@felix_mittermeier">Felix Mittermeier</a> on Unsplash</figcaption>

</figure>

Web development's not easy

When you're working with a team of developers, especially on a team responsible for managing multiple applications, you very well might have JavaScript apps that run on different versions of Node.js.

Some might use Node 10, others Node 12, some may use Yarn as their package manager, others might use npm - and keeping track of all that is really hard. Ensuring every developer on the team is developing with the correct versions all the time is even harder. But it's essential.

While the consequences might be relatively minor during local development: it works on one dev's machine and throws an error on another's, this sort of lack of standardization and clarity can have devastating effects when it comes to production.

I speak from personal experience when I say this sort of thing happened to me and my development team.

We built our app on local machines running Node 10, but the build pipeline defaulted to the lowest Node version it had on hand, Node 6, and the app wouldn't start up in production. We had to roll back the deployment, figure out what went wrong - it turned into a very long night. It went on to live in infamy as "Dark Thursday" for all who experienced it.

And it could have all been avoided if we'd been using a handy little tool called Volta. I want to introduce Volta to you today so you can avoid the stress we went through - it's simple to get started with and can prevent catastrophes like this.

What is Volta?

Volta touts itself as "a hassle-free way to manage your JavaScript command-line tools."

What this means in practice is that Volta makes managing Node, npm, yarn, or other JavaScript executables shipped as part of packages, really easy.

Why Volta?

I've told you what Volta is, but you're probably still wondering why I chose it in particular - it's certainly not the only game in town. NVM's another well known option for managing multiple versions of Node.

NVM

I used to use Node Version Manager (NVM) myself. Heck, I even wrote a whole blog post about how useful it was.

NVM is good, it does exactly what it sounds like: it allows you to easily download and switch versions of Node.js on your local machine.

While it does make this task simpler, NVM is not the easiest to setup initially, and, more importantly, the developer using it still has to remember themselves to switch to the correct version of Node for the project they're working on.

Volta

Volta, on the other hand, is easy to install and it takes the thinking part out of the equation: once Volta's set up in a project and installed on a local machine, it will automatically switch to the proper versions of Node.

Yes, you heard that right. Similar to package managers, Volta keeps track of which project (if any) you’re working on based on your current directory. The tools in your Volta toolchain automatically detect when you’re in a project that’s using a particular version of the tools and takes care of routing to the right version of the tools for you.

From that point on, every time you run Node inside your project directory, Volta automatically switches to that same version of Node you chose.

Not only that, but it will also let you define yarn and npm versions in a project, and if the version of Node defined in a project isn't downloaded locally, Volta will go out and download the appropriate version.

But when you switch to another project, Volta will defer to any presets in that project or revert back to the default environment variables. Cool, right?

Ready to see it in action?

Setup Volta in a project

For ease of getting started, let's create a brand new React application with Create React App, then we'll add Volta our local machine and our new project.

If you'd like to download a working version of this code, I have a GitHub repo here.

Spin up a new React app

First things first, create a new app.

Run the following command from a terminal.

$ npx create-react-app volta-sample-app

Once you've got your new React app created, open up the code in an IDE, and start it up via the command line.

$ npm run start

If everything goes according to plan, you'll see the nice, rotating React logo when you open up a browser at http://localhost:3000.

Now that we've got an app, let's add Volta to it.

Download Volta locally

Installing Volta to your development machine is a piece of cake - no matter your chosen operating system.

Unix

If you're using a Unix based system (MacOS, Linux or the Windows Subsystem for Linux - WSL) to install Volta, it's super easy.

In a terminal, run the following command:

$ curl https://get.volta.sh | bash

Windows

If you've got Windows, it's almost this easy. Download and run the Windows installer and follow the instructions.

Once Volta's finished downloading, double check it installed successfully by running this command in your terminal:

$ volta -v

Hopefully, you'll see a version for Volta like my screenshot below. If you don't try quitting your terminal completely, re-opening a new terminal session and running that command again.

The current version of Volta on my machine is now v1.0.5.

Define your environment variables

Before we add our Volta-specific Node and npm versions to our project, let's see what the default environment variables are.

Get a baseline reading

In a terminal at the root of your project, run the following line:

$ node -v && npm -v

For me, my default versions of Node and npm are v14.18.1 and v6.14.15, respectively.

With our baseline established, we can switch up our versions just for this project with Volta's help.

Pin a node version

We'll start with Node. Since v16 is the current version of Node, let's add that to our project.

Inside of our project at the root level where our package.json file lives, run the following command.

$ volta pin node@16

Using volta pin [JS_TOOL]@[VERSION] will put this particular JavaScript tool at our specified version into our app's package.json. After committing this to our repo with git, any future devs using Volta to manage dependencies will be able to read this out of the repo and use the exact same version.

With Volta we can be as specific or generic as want defining versions, and Volta will fill in any gaps. I specified the major Node version I wanted (16) and then Volta filled in the minor and patch versions for me. Pretty nice!

When you've successfully added your Node version, you'll see the following success message in your terminal: pinned node@16.11.1 in package.json (or whatever version you pinned).

Tip: Make your pinned Node version match your build server's Node version

Had Volta been an option at the time, my team might never have had to go through "Dark Thursday".

Instead, we could have pinned the same Node version used in our build pipeline and production servers to our project for local development to prevent such compatibility issues.

At the very least, we could have realized ahead of time we might have a problem with production before deploying.

Pin an npm version

That was pretty straightforward, now let's tackle our npm version. Still in the root of our project in the terminal, run this command:

$ volta pin npm

In this particular instance, I didn't even specify any sort of version for npm, so Volta defaults to choosing the latest LTS release to add to our project. Convenient.

The current LTS version for npm is 8, so now our project's been given npm v8.1.0 as its default version.

Check your package.json

To confirm the new JavaScript environment versions are part of our project, check the app's package.json file.

Scroll down to the bottom and you should see a new property named "volta". Inside of the "volta" property should be a "node": "16.11.1" and an "npm": "8.1.0" version.

From now on, any dev who has Volta installed on their machine and pulls down this repo will have their settings for these tools automatically switch to use these particular node and npm versions.

To make doubly sure, you can also re-run the first command we did before pinning our versions with Volta to see what our current development environment is now set to.

$ node -v && npm -v

After this, your terminal should tell you it's using those same versions: Node.js v16 and npm v8.

Watch the magic happen

Now, you can sit back and let Volta handle things for you. Just like that. 😎

If you want to see what happens when there's nothing specified for Volta (like when you're just navigating between repos or using your terminal for shell scripts), try navigating up a level from your project's root and checking your Node and npm versions again.

In the screenshot below, I opened two terminals side by side: the one of the left is inside of my project with Volta versions, the one on the right is a level higher in my folder structure.

I ran the following command in both:

$ node -v && npm -v

And in my project, Node v16 and npm v8 are running, but outside of the project, Node v14 and npm v6 are present. I did nothing but switch directories and Volta took care of the rest.

Try and tell me this isn't cool and useful. I dare you. 😉

Conclusion

Building solid, stable apps is tough enough without having to also keep track of which versions of Node, yarn and npm each app runs best with. By using a tool like Volta, we can take the guesswork out of our JavaScript environment variables, and actually make it harder for a member of the dev team to use the wrong versions than the right ones.

And remember to double check your local Node version matches your production server's Node version, too.

If you enjoyed this, you may be interested in checking out my course "The newline Guide to Modernizing an Enterprise React App".

References & Further Resources

Animating React with React Transition Group

Tue, 14 Apr 2020 00:00:00 GMT

import YouTube from '../../src/components/YouTube.astro';

</figure>

Introduction

A few months ago, I gave a talk with a coworker at the tech conference Connect.Tech about the ever-growing need for responsive web design when building websites and applications. During the presentation, we discussed a few different ways to approach it, specifically when it comes to the JavaScript framework React.

While I won't go into all the details in our talk of how to approach responsive design (if you'd like, you can see the full slide deck of the talk here), I did want to share a handy React animation library I stumbled across while building the mobile-responsive demo application, called React Transition Group.

Today, I'll cover what React Transition Group is, the different component options it provides, and how I used it to animate a mobile navbar in a demo application.

React Transition Group

What makes it different?

Unlike other React animation libraries like React Spring or React Reveal, React Transition Group "exposes simple components useful for defining entering and exiting transitions...it does not animate styles by itself. Instead it exposes transition stages, manages classes and group elements and manipulates the DOM in useful ways, making the implementation of actual visual transitions much easier."

React Transition Group is a lower-level type of animation library. It doesn't care nearly as much about what type of animation you'd like to do, it just makes it easier to do any sort of animation on any React component with as little hassle as it can.

And it doesn't hurt to know that React Transition Group began in the original React framework (it's mentioned in the docs) before being spun out into a new package to be maintained by the community. That's a pretty good endorsement for trying RTG, in my book.

Now that you know a little more about RTG's approach to animation, let me cover a few of the different component options it gives users, and how they work.

Types of RTG Components

React Transition Group offers four different types of components for users to choose from based on their animation needs.

Transition

The first component to cover is Transition. This component lets you describe a transition from one component state to another over a span of time with a simple API. Most commonly it's used to animate the mounting and unmounting of a component, but it can also be used to describe in-place transition states as well. Personally, I tend to favor CSSTransition over the straight Transition component, but that's just me.

The transition component tracks the "enter" and "exit" states for the component. The four main states a Transition can be in are:

'entering'
'entered'
'exiting'
'exited'

The transition state is toggled on via the in prop, when it's true the component will begin the entering stage for the duration of the transition until it's fully visible, at which point it will switch to the entered stage. Upon exit, the same things will happen with exiting and exited.

A simple example of fading in a component on enter and fading it out on exit might look something like this:

import { Transition } from 'react-transition-group';

const duration = 300;

const defaultStyle = {
  transition: `opacity ${duration}ms ease-in-out`,
  opacity: 0,
};

const transitionStyles = {
  entering: { opacity: 1 },
  entered: { opacity: 1 },
  exiting: { opacity: 0 },
  exited: { opacity: 0 },
};

const Fade = ({ inProp }) => (
  <Transition in={inProp} timeout={duration}>
    {(state) => (
      <div
        style={{
          ...defaultStyle,
          ...transitionStyles[state],
        }}
      >
        I'm a fade Transition!
      </div>
    )}
  </Transition>
);

In the example above, once the inProp prop is true, the Transition component will activate and begin fading in to view courtesy of the transitionStyles variable (where opacity is defined) that correspond to the various "enter" or "exit" stages I outlined earlier.

It's worth noting this is a platform-agnostic base component -- if you're looking for CSS transitions (like I was for my use case), you'll probably want the CSSTransition component instead.

CSSTransition

If you're using CSS transitions or animations, the CSSTransition component is what you'll want to use; it's built upon the Transition component, so it inherits all of its props.

To work, CSSTransition applies a pair of class names during the appear, enter, and exit states of the transition. The first class is applied and then a second *-active class in order to activate the CSS transition. After the transition, matching *-done class names are applied to persist the transition state.

For instance, if your CSSTransition component's classNames property is sample, you'd first see sample-enter, then sample-enter-active, and finally sample-enter-done for the ending state when the animation is done. And when it's time to reverse the animation (such as with a modal or slider that needs to disappear or exit the screen), you'd see it cycle through sample-exit and sample-exit-active and sample-exit-done on your component's classNames property.

Here's a simplified code sample of what the JSX and accompanying CSS classes might look like for making a div of text fade in or out. (I've omitted the imports at the top of the file because they're the same named import for each type of component from RTG: import { CSSTransition } from 'react-transition-group';.)

First the JavaScript code showing the CSSTransition component wrapping a div element to show or hide based on the state of isVisible, which is toggled by the button element underneath.

const FadeExample = () => {
  const [isVisible, setIsVisible] = useState(false);

  return (
    <>
      <CSSTransition
        in={isVisible}
        timeout={300}
        classNames="fade"
        unmountOnExit
      >
        <div className="fade-text">I'll fade in and out</div>
      </CSSTransition>
      <button onClick={() => setIsVisible(!isVisible)}>
        Toggle Visibility
      </button>
    </>
  );
};

And here's the CSS classes that the CSSTransition component would experience once isVisible became true and triggered the in prop to fade the text in or out (just like with the original Transition component).

.fade-enter {
  opacity: 0;
}

.fade-enter-active {
  opacity: 1;
  transition: opacity 300ms ease-in;
}

.fade-enter-done {
  opacity: 1;
}

.fade-exit {
  opacity: 1;
}

.fade-exit-active {
  opacity: 0;
  transition: opacity 300ms ease-in;
}

.fade-exit-done {
  opacity: 0;
}

*-active classes represent which styles you want to animate to, so it's important to add the transition declaration only to them, otherwise transitions might not behave as intended. This might not be obvious when the transitions are symmetrical, i.e. when *-enter-active is the same as *-exit, but it becomes apparent quickly in more complex transitions.

Let's move on now to React Transition Group's next option: SwitchTransition.

SwitchTransition

This component is useful if you want to control the render between state transitions, it's inspired by Vue transition modes. Based on the selected mode (in-out or out-in), and the child's key which is the Transition or CSSTransition component, the SwitchTransition makes a consistent transition between them.

If out-in mode is selected, SwitchTransition waits until the old child leaves and then inserts a new child. If the in-out mode is selected, the SwitchTransition inserts a new child first, waits for the new child to enter and then removes the old child.

Here's an example so you can see how the code might be structured to make two different buttons "switch" places in the DOM based on the current state. Once again, imports at the top of the file omitted for brevity.

const SwitchExample = () => {
  const [state, setState] = useState(false);

  return (
    <SwitchTransition mode="out-in">
      <CSSTransition
        key={state ? "Goodbye, world!" : "Hello, world!"}
        addEndListener={(node, done) =>
          node.addEventListener("transitionend", done, false)
        }
        classNames="fade"
      >
        <button onClick={() => setState((state) => !state)}>
          {state ? "Goodbye, world!" : "Hello, world!"}
        </button>
      </CSSTransition>
    </SwitchTransition>
  );
};

Here, SwitchTransition wraps the CSSTransition component, which actually handles the logic and animating of entering a new Button and exiting the existing one. The key in CSSTransition keeps track of state in the component, and every time the state changes (based on a button click), the CSSTransition component takes action and SwitchTransition handles keeping the old button visible until after the new button has appeared, which CSSTransition is fading out based on its event listener.

That addEndListener function is crucial to animating the switch between elements, without it, the state (and components) will instantaneously flip like there's no animations at all.

Here's the CSS for the fade class name associated with CSSTransition.

.fade-enter {
  opacity: 0;
}

.fade-exit {
  opacity: 1;
}

.fade-enter-active {
  opacity: 1;
  transition: opacity 200ms;
}

.fade-exit-active {
  opacity: 0;
  transition: opacity 200ms;
}

The CSS here is really concerned with the CSSTransition component wrapped inside of SwitchTransition, but I figure the more sample code you see for how to animate things in and out of the DOM, the better.

It takes some trial and error to get it right, but the documentation for SwitchTransition is good and I'm sure you'll figure it out without much of a problem if this is the kind of animation you desire in your application.

TransitionGroup

Last but not least is the <TransitionGroup> component. This component manages a set of transition components (<Transition> and <CSSTransition>) in a list. Like with the individual transition components, <TransitionGroup> is a state machine for managing the mounting and unmounting of components over time.

Please note that <TransitionGroup> does not define any animation behavior. Exactly how a list item animates is up to the individual transition component, which means you can mix and match animations across different list items, which could be handy.

Unlike SwitchTransition which lets you control the entering and exiting of elements in the DOM, TransitionGroup will make the animations happen simultaneously (i.e., to remove the old child and insert a new child at the same time).

Take a look at this example of a simple chore list that can have chores removed from it once they're completed. First, the JSX code with no imports in the example.

const ChoreList = () => {
  const [chores, setChores] = useState([
    { id: 1, text: "Do laundry" },
    { id: 2, text: "Wash dishes" },
    { id: 3, text: "Vacuum" },
    { id: 4, text: "Walk the dog" },
  ]);

  const removeChore = (id) => {
    setChores((chores) => chores.filter((chore) => chore.id !== id));
  };

  return (
    <TransitionGroup className="chore-list">
      {chores.map(({ id, text }) => (
        <CSSTransition key={id} timeout={500} classNames="chore">
          <div>
            <span>{text}</span>
            <button onClick={() => removeChore(id)}>Done</button>
          </div>
        </CSSTransition>
      ))}
    </TransitionGroup>
  );
};

And, here's the CSS that will make the chores fade out of the DOM as they're removed.

.chore-enter {
  opacity: 0;
}

.chore-enter-active {
  opacity: 1;
  transition: opacity 500ms ease-in;
}

.chore-exit {
  opacity: 1;
}

.chore-exit-active {
  opacity: 0;
  transition: opacity 500ms ease-in;
}

And with that, I've covered all the main component options React Transition Group offers. All in all, using one or more of these components together gives you fine grained control over what type of and how your React components animate in the DOM.

Now, let's get down to the business of how exactly I used React Transition Group to animate my mobile navbar in my demo site.

How I used RTG in My React App

The use case I had for React Transition Group is a fairly standard one, I'm sure.

For my app, a movie demo site I built using the Movie Database API for a data source, when it was in mobile view, I wanted the navbar (normally across the top of the page in larger layouts) to condense down to the well-recognized hamburger menu, and require users to click it to open the list of links.

What I needed was to animate a mobile-sized navbar to slide into and out of the DOM for the user, at the touch of a button. Seemed like a simple enough request.

Here's a video below of what I envisioned happening.

Easy enough, right? Slide a navbar in when the user clicks the hamburger icon. slide it back out when they click the icon again.

The code I'll be showing you are actual snippets taken from the demo site you're seeing in the video above. If you'd like to see the whole codebase for this project, it's available on GitHub, and if you'd like to see a live version of the site, it's available on this website (just make your desktop browser mobile-sized to see the animated navbar).

Step 1: Add CSSTransition Group to the Sidebar.js Component

The first thing I had to do after installing React Transition Group into my project's dependencies with npm -i react-transition-group, was add it to the component I wanted to animate.

This component is named Sidebar and its full JavaScript code looks like this.

Sidebar.js

import React, { useState } from "react";
import { CSSTransition } from "react-transition-group";
import { Link } from "react-router-dom";
import "./Sidebar.scss";

const Sidebar = () => {
  const [expandedLinks, setExpandedLinks] = useState(false);

  const toggleLinks = () => {
    setExpandedLinks(!expandedLinks);
  };

  return (
    <div className="sidebar-container">
      <div className="sidebar-hamburger" onClick={toggleLinks}>
        <div className="hamburger-line" />
        <div className="hamburger-line" />
        <div className="hamburger-line" />
      </div>
      <CSSTransition
        in={expandedLinks}
        timeout={350}
        classNames="visible"
        unmountOnExit
      >
        <div className="sidebar-links" onClick={toggleLinks}>
          <Link to="/" className="sidebar-link">
            Popular
          </Link>
          <Link to="/top-rated" className="sidebar-link">
            Top Rated
          </Link>
          <Link to="/now-playing" className="sidebar-link">
            Now Playing
          </Link>
          <Link to="/upcoming" className="sidebar-link">
            Upcoming
          </Link>
        </div>
      </CSSTransition>
    </div>
  );
};

export default Sidebar;

I'll quickly break down what goes into this component. The way that Sidebar triggers the in prop in the CSSTransition component (which will reveal all the navigation links to other pages in the site housed within the sidebar) is with the toggleLinks function.

Once toggleLinks changes the expandedLinks state to true, the CSSTransition component runs and adds the class name modifiers ( enter, enter-active, exit, etc.) to the visible class present on the CSSTransition component.

And all I had to do to hide the sidebar links until I'm ready for them, was wrap the main div surrounding all the links with the CSSTransition component. Pretty straightforward, so far, right?

Step 2: Generate the CSS to Support RTG's Animation Class Names

Now for the CSS. It's long, I'll admit -- I did a lot of styling to make this navbar look just how I wanted, but try to bear with me. It's really not complicated, just verbose from styling (and this is a condensed version where I cut out some extraneous CSS not relevant to this article).

Sidebar.scss

.sidebar-container {
  display: flex;
  flex-direction: column;
  position: relative;

  .sidebar-hamburger {
    display: flex;
    flex-direction: column;
    cursor: pointer;
    padding: 10px;

    .hamburger-line {
      width: 25px;
      height: 3px;
      background-color: white;
      margin: 2px 0;
    }
  }

  .sidebar-links {
    display: flex;
    flex-direction: column;
    position: absolute;
    top: 45px;
    left: 0;
    background-color: #333;
    width: 200px;
    z-index: 100;

    .sidebar-link {
      padding: 10px 20px;
      color: white;
      text-decoration: none;
    }
  }

  .visible-enter {
    transform: translateX(-100%);
  }

  .visible-enter-active {
    transform: translateX(0);
    transition: transform 0.3s linear;
  }

  .visible-enter-done {
    transform: translateX(0);
  }

  .visible-exit {
    transform: translateX(0);
  }

  .visible-exit-active {
    transform: translateX(-100%);
    transition: transform 0.3s linear;
  }
}

If you wade through all this styling and focus on the transform and translate properties in each class I've defined above, you'll see what is causing the animation I want. I slide the navbar in to view on .visible-enter-active with transition: transform 0.3s linear; and slide it back out of view with .visible-exit-active and transition: transform 0.3s linear;. Piece of cake!

Conclusion

Responsive web design is only gaining more importance as technology touches more and more of our daily lives, and smooth animations to make that technology feel more natural are essential.

React Transition Group makes animations of all kinds easy by providing 4 basic components and an easy-to-use API unconcerned with the specific types of animation it's enabling. This low-level implementation gives developers fine-grain control over exactly how their app's components flow in the DOM for the best user experience they can provide. RTG is really easy to get up and running, and a pleasure to develop with.

Check back in a few weeks -- I'll be writing more about JavaScript, React, ES6, or something else related to web development.

Thanks for reading. If you're looking to add some slick animations to your React app, I hope you'll consider React Transition Group among the myriad of libraries out there. It makes animations simple and straightforward, even down to the tiny, micro-animation details. Give it a shot -- you won't regret it.

References and Further Resources

GitHub repo, React app with using React Transition Group
Live demo site of responsive app
Responsive design featuring React.js presentation deck
React Transition Group, site documentation

9 React Testing Library Tips and Tricks

Wed, 01 Apr 2020 00:00:00 GMT

</figure>

A few weeks ago, I had the good fortune to attend a two-day training course at work on improving my React Testing Library knowledge and skills.

At my job, my team relies heavily on many forms of automated testing (unit, integration, end-to-end, etc.) to help ensure that all the pieces of our app are working correctly. So when I took the course, I already had a basic understanding of how to use React Testing Library (RTL) because we had begun using it a few months prior as we started writing more React hooks-based components in our application. Our previous unit testing combination of Jest and Enzyme had stopped meeting our needs as well.

This training, however, opened my eyes to some really helpful info around using RTL I was totally unaware of. I wanted to share nine of the biggest takeaways for me in hopes that they'll help you in your own use of this testing library.

What Is React Testing Library? And Why Is It Different?

Note: This is not a "Getting Started with RTL" piece. This article assumes you already have a pretty good understanding of how to use the testing framework. My aim is to show you some additional tricks to make writing (and debugging) tests easier.

With that said, I will give a brief overview of the whys and hows of React Testing Library before I dive into the meat of this article.

The why

Kent C. Dodds created the React Testing Library to address one problem: other testing libraries encouraged developers to write tests that were too tightly coupled to the implementation details of the components they were testing. Dodds believed that the tests should only verify the end results (the behavior) of the components.

It makes sense when you think about it, right?

Why don't we write integration tests closer to the way we write end-to-end tests? Tests that focus on the user's perspective (how the component looks and behaves in the DOM). Because that's what ultimately matters in the end: Do the components do what they're supposed to do when a user interacts with them?

We don't necessarily care if it stores something in the Redux store or makes a particular API call to fetch data. Users don't know or care about that -- they care about if the screen shows the info they clicked on or the item they ordered is being shipped to them. And as developers, RTL's stance is that we should care about that too.

To that end, Dodds recommends approaching testing in the following way:

Avoid dependencies on the implementation details of your components.
Make testing easy to maintain (i.e. refactoring the component implementation does not break the tests).
Avoid shallow mounting, as we want to test that the components integrate to solve a user problem (such as editing data in a form).

And this leads me to...

The how

The React Testing Library is a very lightweight solution for testing React components. It provides light utility functions on top of react-dom and react-dom/test-utils in a way that encourages better testing practices.

Here are some interesting aspects of React Testing Library:

It focuses on verifying actual DOM nodes rather than verifying instances of rendered React components.
It provides utilities for querying the DOM in the same way the user would.
It finds elements by their label text (just like a user would).
It finds links and buttons from their text (like a user would).
It provides a way to find elements by a data-testid as an "escape hatch" for elements where the text content and label do not make sense or are not practical or available.

Now that that's out of the way, let's get to the good stuff: the hot tips.

Tip #1: Screen Instead of Component

Have you ever thought to yourself, "Boy, writing component.getByText... or component.getByLabel... all the time sure is annoying. I wish there was another way"? Turns out there is: It's called screen.

All of the queries exported by DOM Testing Library (which RTL is built on top of) accept a container as the first argument. Because querying the entire document.body is very common, DOM Testing Library also exports a screen object that has every query that is pre-bound to document.body (using the within functionality).

Here's how you use it:

import React from "react";
import { screen } from "@testing-library/dom";
import { render } from "@testing-library/react";
import { act } from "react-dom/test-utils";
import App from "./App";

describe("Product Browser App", () => {
  it("renders a welcome message", () => {
    act(() => {
      render(<App />);
    });
    screen.findByText("Hello world");
  });
});

Trust me: After a while, screen will become increasingly handy as you're writing tests to check what's visible or enabled (or not) in the DOM as you manipulate it.

Tip #2: FindByText Instead of waitForElement...getByText

If you've used RTL before, you probably know it has several handy async functions that start with waitFor, waitForElement, or waitForElementToBeRemoved. These can be useful to wait for an element to appear or disappear in response to an action.

Typically, the waitFor code would look something like this:

const orderButton = await waitForElement(() => container.getByTestId("order-submit-button"));

The line above is pretty clean and concise, but there's an even cleaner way to write this: with the help of findBy....

Check this out:

const orderButton = await container.findByTestId("order-submit-button");

By using any of the findBy queries that RTL makes available (findByText, findByTestId, findByPlaceholder, etc.), it automatically wraps the element to be searched for in a promise that will wait for it to be rendered, saving you from having to write the wrapper code yourself.

On its own, this might not seem like much, but imagine writing many multiples of this sort of test all over your application. The leaner test code starts to add up quickly.

Tip #3: UserEvent for Async Actions

H/T to Kent C. Dodds for letting me know that functions like render() and fireEvent() don't actually need to be manually wrapped in act() -- every async utility is automatically wrapped courtesy of RTL. Sweet -- that's even more knowledge for you!

I do want this to stay a 9 tip blog post, however, so I still wanted to share a library making asynchronous actions like click() or select() just a wee bit simpler, just so you're aware it exists.

Here's what the typical async event code looks like in RTL:

await fireEvent.click(orderButton);

Not the worst thing to have to write... but consider the amount of async / await events you have for one fairly complex test case. And write multiple tests to test one component.

The user-event library "tries to simulate the real events that would happen in the browser as the user interacts with it. For example userEvent.click(checkbox) would change the state of the checkbox."

Just import it at the top of your test file like so:

import userEvent from "@testing-library/user-event";

Now, here's that same code from above with the userEvent library applied:

userEvent.click(orderButton);

Look at how that code got even leaner! This, to me, was a game-changer when I learned about it and I absolutely love using it now in my own tests.

Note: The user-event library doesn't fully support all async actions in RTL yet. Some actions like change and keyDown are not available in user-event, but the library is looking for people to add to it, so if you'd like to contribute, I'm sure they'd appreciate the help.

Tip #4: PrettyDOM Instead of debug

Debugging non-working code is painful. Debugging non-working tests can be even more painful -- especially when you run RTL's component.debug() on the component, but the component's DOM tree is so large it cuts off before the whole thing is rendered in the console.

console.log(component.debug());

Here's what the console shows when the typical debug() runs in an RTL test:

<figcaption>Notice the "..." at the end of this screenshot? That's where the render of the DOM tree in the console ends.</figcaption> </figure>

There's a solution to this annoyance, however. It's called prettyDOM(), and RTL's debug() method is actually built on top of it. The prettyDOM function originates in the DOM Testing Library and allows for extra configuration beyond just specifying the component or element to print out. It has an optional maxLength parameter to limit the size of the resulting string -- or, in my case, to print out very, very large DOM trees.

Here's how I use prettyDOM to print those pesky extra-big DOM trees:

console.log(prettyDOM(component.container, 99999));

And here is another snippet of the DOM tree being rendered to the console with prettyDOM. Notice that it keeps going after the RTL debug() method ends:

<figcaption>This rendering of the DOM tree goes on for a lot longer, but you can see it keeps going past the ".ant-radio-button-checked" span where it cut off before.</figcaption> </figure>

Lately, prettyDOM's been a real life-saver for me when I've been trying to debug some particularly tricky components in my team's application. Without this option, it would have taken me so much longer to figure out what the problems with the test were.

Tip #5: Document.querySelector to Check a Component's Classes

I hope you don't run into this issue -- I really do -- but if you're using a component library in your React application like Ant Design, you're going to learn very quickly that testing some of the more complex components (e.g. tables, radio buttons, etc.) can get really painful with RTL.

This is because a lot of the typical ways you'd use to ID and check elements with RTL don't exist (or are too generic) on Ant components. And in some cases, you need to check whether Ant elements are enabled or disabled, but unlike how a custom component might have a data-testid to target, Ant does not. It has an additional class added to its className property instead.

But RTL doesn't offer an option for checking component classes. In that case, we have to resort to the document level of the DOM and employ document.querySelector.

To check if an Ant component is hidden, here's how I did it below:

expect(document.querySelector(".hidden")).toBeInTheDocument();

It's not particularly elegant code, but it works in a pinch. Just like Kent C. Dodds says to consider data-testid as an "escape hatch," consider document.querySelector() in the same vein. Use it sparingly and only if you have no other options to target an element in RTL.

Tip #6: ParentElement to Access Elements Above the Target

This is a lesser-known trick I only learned this past week -- again because Ant components are a pain to test with React Testing Library.

I needed to enter text into an input, then click a plus (+) button at the end of the input box for the value to be saved into the DOM. The plus button, however, happened to be a level above the input element in the component, so in order to target it, I learned about the parentElement selector, which is actually native to the browser, not RTL specifically. And I couldn't just search the whole component for the plus button because there were multiple plus buttons on the page for multiple inputs.

const input = component.queryByPlaceholderText(componentText);
const enter = input.parentElement.querySelector(".anticon-plus");

As you can see in the example above, first I created the variable input targeting the input, then I created the second variable enter targeting the plus button in the input's parent element.

This let me click the correct input plus button even though there was no data-testid or other individually defining characteristic about this Ant input. It's come in pretty handy as of late. I hope it can help you too.

Tip #7: PreviousSibling and nextSibling to Access Elements in Line With the Target

In a very similar situation, I also learned about the options of previousSibling and nextSibling (also browser native properties) to select elements around a particular element being targeted in the DOM by React Testing Library.

In this scenario, I needed to click a checkbox in an Ant table component to select one of the options present in the table. The checkbox element was a sibling of the option's name, which was the only unique element present in each row of the table.

const keyToSelect = await component.findByText("key to select");
userEvent.click(keyToSelect.previousSibling.firstChild.firstChild);

First, I created a variable (keyToSelect) targeting a single row in the Ant table using the unique name of the row ("key to select"). Then I was able to use the previousSibling of the targeted element to access the checkbox and select the row inside the table.

Not exactly the easiest syntax, I'll admit, but it solved my problem. And you can do exactly the same type of element targeting using nextSibling as well.

Tip #8: Use { exact: false } for Fuzzy Text Matches in the DOM

Sometimes, it's tough to use one of the getByText- or findByText-type searches in RTL because it defaults to exact matching. This means getByText("Hi there") will only work if the string is exactly the same in the DOM: full-string, case-sensitive. If there's a <span>, a newline (\n), or something else between "Hi" and "there", this getByText() query would fail to find the text.

To do a fuzzy match, which will not be case-sensitive and will include substrings, you can add { exact: false }.

Check out this example below:

<div>Hi there!</div>

getByText(container, "i the", { exact: false });
getByText(container, "hi there", { exact: false });

Although it's more recommended to use a regex if you need to match a substring, for a simpler text selection in the DOM, I think this is a fine substitute. And I don't know about you, but I always have to test regex out in the browser to remember how to write it. I use it so infrequently.

Tip #9: Open Any File in the Browser From the Command Line

Final tip, and this one isn't actually specific to React Testing Library, but I learned it in this class as well and I think it's pretty cool.

If you're using VS Code as your JavaScript IDE of choice and you want to open a file in the browser from the VS Code built-in terminal command line, it's a snap.

In the command line, all you have to do is type open <file path>, and voila: A new window pops open in your browser.

The reason I mention it here is because it provided a quick and easy way for me to check my code coverage, as I added new tests to our application. I find it a lot easier to click around in the browser version of code coverage to see which lines are tested than using the little printout in the terminal.

For example, to open code coverage for our project, I'd write:

open coverage/lcov-report/index.html

And there it is, ready to be examined in detail. VS Code handy shortcuts for the win. I thought it was a fun fact and I hope you might find it useful too.

Conclusion

React Testing Library is a powerful framework that's really changed the way we approach integration testing React applications. It took a minute for me to wrap my head around the changes that RTL entails, but I think it makes more sense for how the application should really work in the real world.

And although I knew some things about React Testing Library, I learned a lot of lesser-known tips and tricks that have helped simplify and streamline my own testing.

Check back in a few weeks -- I'll be writing more about JavaScript, React, ES6, or something else related to web development.

Thanks for reading. I hope you learned a few new things that will make your own use of the React Testing Library that much easier. Integration testing nowadays is a necessary piece of any good web application. The least I can do is try to make it a little easier for you.

References and Further Resources

React Testing Library documentation
React Testing Library GitHub repo
RTL User Event GitHub repo

How to Use React's Context API and useContext() Hooks Effectively

Tue, 03 Mar 2020 00:00:00 GMT

</figure>

Introduction

At work recently, I was faced with a problem that's quite common, thanks to React's one-way data flow. At a high level of my application, I opened a checkout drawer that contained a user's selection of items to order. In another, much lower component, I triggered the actual ordering of those items. Once the order was successfully placed, I needed the drawer to automatically slide closed again.

Following so far?

Because of the nature of my app's checkout process, there were a lot of intermediate components between the top-level, parent component (controlling the drawer's open or closed state), and the final checkout child component inside the drawer (containing the button and function to place the order).

I wanted to avoid having to pass down props containing the drawer's open state and a function to toggle the drawer's state back to closed upon order success. None of the other intermediate components needed those props for anything, so it would be extra code that didn't need to be there. Then I lit upon a solution — something I'd heard a bit about but had never had a need to use: React's Context.

Today, I'll be detailing how I used React's Context API and its useContext hook to avoid prop drilling both application state and functions.

What's the React Context API?

Before I get too far into the weeds of how I employed the Context API to help solve my problems, let me go over what it is and how it works, so we're all on the same page.

React's official documentation does the best job of defining Context:

Context provides a way to pass data through the component tree without having to pass props down manually at every level. — React Docs

Just as in the scenario I described above, Context exists in React as a way to pass data around an application without having to explicitly pass a prop through every level of the tree. It's React's built-in way of sharing data that can be considered "global" for a tree of React components.

Examples of this could include user authentication, preferred language or theme setting, or, as with my situation, a child component telling a parent component to switch its state from an open drawer to a closed drawer.

Yes, you could use Redux for this sort of global state management as well, but the boilerplate code required to set up Redux, its actions and its dispatchers, is a lot of extra work for what should be a relatively simple set of tasks. Plus, with the release of React Hooks, using context has been made easier than before.

OK, let's talk about how to use React's Context API and the useContext Hook. I'll show you how to implement it using my own application.

Using React Context

The first thing you need to do to use the Context API is to create a new Context file. This is the file that will hold any data variables, default values, and functions you might want to make accessible to the components when using this context.

React.createContext()

Inside your context file, which I tend to name XyzContext.js, you'll create this particular instance of Context, and assign it some default values.

Here's the context file I made for my application to handle the visibility of the checkout drawer and the toggle of its visibility state.

CheckoutDrawerContext.js

import { createContext } from 'react';

const CheckoutDrawerContext = createContext({
  showDrawer: false,
  toggleCheckoutDrawer: () => {},
});

export default CheckoutDrawerContext;

As you can see from my code snippets, to create a new instance of Context is relatively easy. In my case, after calling creatContext(), I just passed in an object with a boolean about the checkout drawer's current visibility (showDrawer with a default starting value of false) and an empty arrow function called toggleCheckoutDrawer(), which will eventually be associated with a function that actually affects the drawer's state of being shown or hidden.

When React renders a component that subscribes to this CheckoutDrawerContext object it will read the current context values from the closest matching Provider above it in the tree.

The default value argument (showDrawer, in this case) is only used when a component does not have a matching Provider above it in the tree.

Now that this particular instance of Context is created, we can move on to using it in my application.

Context.Provider and its values

Every Context object comes with a Provider React component that allows consuming components to subscribe to context changes.

The provider accepts value props to be passed to consuming components that are descendants of this Provider. What's also good to know is that one Provider can be connected to many consumers. If multiple children need to know or change the values of the provider, they can.

Cart.js

<CheckoutDrawerContext.Provider value={{ showDrawer, toggleCheckoutDrawer }}>
  <section className="cart-checkout">
    <CartDrawer selectedCartItems={selectedCartItems} />
  </section>
</CheckoutDrawerContext.Provider>

For my application, inside of my top level Cart component, I wrapped my CartDrawer component in my CheckoutDrawerContext.Provider. I did this because child components nested within CartDrawer would need access to the values the CheckoutDrawerContext.Provider contains — in this case, the showDrawer state and the toggleCheckoutDrawer() function.

As for the values being passed to the CheckoutDrawerContext.Provider, I made them inside the Cart.js file too.

Cart.js

const [showDrawer, setShowDrawer] = useState(false);

const toggleCheckoutDrawer = () => {
  setShowDrawer(!showDrawer);
};

Using the useState() React Hook inside of the Cart component file, I created a piece of state called showDrawer with its setter to update the boolean setShowDrawer. I created a function called toggleCheckoutDrawer(), which would switch the value of showDrawer to whatever its opposite value was.

I named these states and functions after the default values in my CheckoutDrawerContext file, but this is not necessary — I did it only for my own convenience.

Right, so now the Context.Provider is up and running and it has its values of showDrawer and toggleCheckoutDrawer(). I'm ready to move on to the consuming values part.

React.Consumer and React.useContext()

Initially, when Context was first released with React, it was only available to stateful, class-based components.

The syntax for Context consumers is similar to that for Providers to add it to a project — wrap the consumer around the code that needs access to the values, as shown in the sample code below.

ClassBasedChildComponent.js

<SampleContext.Consumer>
  {value => /* do something with the context values passed here */}
</SampleContext.Consumer>

But since my application is using React hooks and the checkout flow is written using functional components, I leveraged the useContext() hook for my components that needed access to the CheckoutDrawerContext instead. In my opinion, it's cleaner code and easier to understand.

CheckoutSummaryDetails.js

import React, { useContext } from 'react';
import { orderItems } from '../../../../services/orderService';
import CheckoutDrawerContext from '../../../../context/CheckoutDrawerContext';

export const CheckoutSummaryDetails = props => {
  const { summaryDetails, userInfo, items } = props;
  const checkoutDrawer = useContext(CheckoutDrawerContext);

  const placeOrder = async () => {
    const executionResponse = await orderItems(userInfo, items);
    if (executionResponse === 'successfully placed order') {
      checkoutDrawer.toggleCheckoutDrawer();
    }
  };

  return (
    <div className="cart-drawer__summary">
      <strong className="cart-drawer__header">Summary</strong>
      <div className="cart-drawer__summary-content">
        <div className="checkout-summary">
          <span>Final Checkout Summary</span> <span>{summaryDetails}</span>
        </div>
      </div>
      <button type="button" className="order-button" onClick={placeOrder}>
        Order Now
      </button>
    </div>
  );
};

The component above, CheckoutSummaryDetails.js, is a very stripped-down version of the actual checkout in my application, but it serves its purpose for this example.

First, note that the CheckoutDrawerContext is imported into this CheckoutSummaryDetails.js file. To actually gain access to the values of this Context, I create a new constant (checkoutDrawer), and wrap the named Context in the useContext() Hook. This gives me access to values within the CheckoutDrawerContext. The rest is fairly straightforward.

When the user clicks the button titled "Order Now," the placeOrder() function is fired in the component and it triggers the orderItems() call to an API or a database or wherever else it might need to send the user info and the items being ordered. Upon a success message of the order being placed, the component that closes the CartDrawer is triggered via checkoutDrawer.toggleCheckoutDrawer(). This will switch the state of showDrawer to some four or five levels higher up this component tree, from true back to false, and the CartDrawer component will be hidden from view again.

That's it! React's Context has prevented unnecessary prop passing and allowed a child component to update the state of a parent component. Sweet!

That wasn't so hard now, was it? Believe me, once you've used Context successfully once, implementing it in lots of other places will be a cinch.

Conclusion

React's Context API and useContext Hook are fairly easy to understand in theory. They're a little less clear when it comes to implementing them the first time around.

But they're such a help. Passing unneeded data through several levels of a component tree when only the second and sixth level of child components actually need those props is not helpful. It just means more props to keep track of, even when all the intermediate components don't need that data.

Yes, Redux could do the same thing, but that's a lot of extra setup for even the simplest pieces of data. And unlike Redux, Context is native to React — it's already built-in, it's ready to go, and once you start using it, it's pretty intuitive. I was hesitant to try it at first, but once I figured it out, I became a big fan. I think you will be too.

Check back in a few weeks — I'll be writing more about JavaScript, React, ES6, or something else related to web development.

Thanks for reading. I hope I've demystified React's Context API and useContext Hook so you can successfully use it in your own React project. You won't regret the cleaner code that it makes possible!

References and Further Resources

React Context API documentation
React Hooks useContext documentation

JavaScript’s Optional Chaining Proposal

Sun, 16 Feb 2020 00:00:00 GMT

</figure>

Introduction

If you’ve been a JavaScript developer for any length of time, you’ll understand the annoyance I’m describing as soon as I set up this scenario.

Picture it: you’re coding away, building something cool, fetching data and manipulating objects, and suddenly you see the error, the dreaded undefined error. And your application breaks. Hard. 😑

<figcaption>The undefined error every JavaScript developer ever, has encountered.</figcaption>

</figure>

You sigh, roll your eyes 🙄, and add yet another check into your if / else statement to prevent undefined errors like this in the future (in this case it’s making sure the genres array has a length of more than 0).

if (!loading && !error && genres.length) {
  // do something here
} else { 
  // handle if something evaluates to not true in the if statement
}
// program continues on its merry way

But what if there was a better way?

What if, instead of checking that several objects or object properties existed before finally getting to what you want to do with that data, the good stuff, the interesting stuff, you could move on to that part even if things did evaluate to null or undefined? If your code didn’t just throw up as soon as it hit an undefined value, instead of what it was expecting?

Sounds like the stuff of dreams? Too good to be true? Not anymore.

Because JavaScript wants to do just that.

That’s what I want to talk about today: the optional chaining operator ?. proposal, which lets us read the value of a property located deep within a chain of connected objects without having to expressly validate that each reference in the chain is valid.

Optional Chaining Operators

As I alluded to in the introduction, the optional chaining operator is a proposal, which is currently in Stage 4, allowing for a sort of short-circuiting if a reference evaluates to null or undefined and returns a value of undefined instead of throwing an error the program cannot continue on from.

This boils down to shorter and simpler expressions when accessing chained properties when the possibility exists that a reference may be missing. It can also be immensely helpful while exploring the contents of an object when there’s no guarantee as to which properties are required.

Syntax of the optional chaining operator

I know this sounds a little complicated in writing, but seeing it in code should make it clearer.

Optional chaining with object properties

Here’s an example of optional chaining when accessing an object’s properties:

const simpleObj = {
  levelOne: {
    levelTwo: "hello world"
  }
};
let nestedProp = simpleObj.levelOne?.levelThree;
console.log("There is a not a levelThree prop, it is ", nestedProp); // prints: "There is a not a levelThree prop, it is  undefined"
console.log("But the program still goes on."); // prints: "But the program still goes on."

The chaining operator is actually made up of just ?.. If the operand at the left-hand side of the ?. operator evaluates to undefined or null, the expression evaluates to undefined. Otherwise the targeted property access, method or function call is triggered normally.

And you can add it to:

An object: a?.b
An object expression: a?.[x]
An array: a?.[index] or
A function’s arguments a?.b().

And it can simplify accessing values through connected objects when it’s possible that a reference or function may be undefined or null.

Before optional chaining, everything had to be validated

Just think about how often you’ve been working with objects that have properties nested more than one level deep, and you’ve had to validate every. Single. One.

Without optional chaining, looking up a deeply-nested sub property meant validating the references in between, like so:

let newSubProperty = obj.prop && obj.prop.subProp;

The value of obj.prop is checked to be non-null and not undefined and then the value of obj.prop.subProp is accessed. This prevents the error that would have happened by attempting to access obj.prop.subProp without first checking that a value for obj.prop existed.

I don’t know about you but this felt like a lot of extra, redundant code to me.

With the optional chaining operator (?.), however, you don't have to explicitly test and short-circuit based on the state of obj.prop before trying to access obj.prop.subProp:

let newSubProperty = obj.prop?.subProp;

By using the ?. operator, JavaScript knows to implicitly check that obj.prop isn’t a null or undefined value before attempting to access the nested subProp. If obj.prop does end up evaluating to either null or undefined, the expression short-circuits and returns an undefined value, and the program moves on.

Let’s look at some more examples of the optional chaining operator in action.

Optional chaining with function calls

You can use optional chaining when attempting to call a method which may not exist. This can be helpful when using an API in which a method might be unavailable, either due to the age of the implementation or because a feature isn’t available on the user’s device.

let response = someInterface.customFunction?.();

By using optional chaining with function calls, the expression will automatically return undefined instead of throwing an exception if the method isn't found.

When using optional chaining with expressions, if the left operand is null or undefined, the expression just won’t be evaluated.

let possibleNull = null;
let value = 0;
let result = possible?.[value++];

console.log(value); // 0 as value was not incremented

Even though the variable possibleNull was indeed null, the variable result didn’t throw an error because the optional chaining operator kicked in. It just skipped over the function and moved on. Useful, no?

Optional chaining with object expressions

You can also use the optional chaining operator when accessing properties with an expression using the bracket notation of the property accessor. Check this out.

let nestedProp = myObj?.["prop" + "Name"];

Being able to use this with bracket notation might is useful in cases where you have a dynamic key. In my opinion, it's a little harder to read that dot notation, but it’s there if you need it.

Array item access with optional chaining

Arrays with optional chaining are interesting, by using the optional chaining syntax you can attempt to access items within an array by the index you pass, and if that index doesn’t exist within the array it won’t error out.

let arr = [1,2,3];
let arrayItem = arr?.[3];
console.log(arrayItem) // prints: undefined

Stacked optional chaining

Another cool thing to note is that you can use optional chaining multiple times within the same nested object.

let customer = {
  name: "Sean",
   details: {
    age: 43,
    location: "Trinidad" // detailed address and subscription service frequency is unknown
  }
};

let customerSubscription = customer.details?.subscription?.frequency;
console.log(customerSubscription); // prints: undefined

let customerCity = customer.details?.address?.city;
console.log(customerCity); // prints: undefined

In both variables, customerSubscription and customerCity, the value ends up being undefined because neither is defined on the customer object. But once again, an error isn’t thrown if this is the case.

You can do this same thing with functions as well: multiple optional chaining operators within function calls. Like so:

let duration = vacations.trip?.getTime?.();

Combining with the nullish coalescing operator

This is pretty useful on its own but the real power comes when optional chaining is combined with the nullish coalescing operator.

I won’t go in to great detail about this feature (also in Stage 4 proposal), but in a nutshell, the nullish coalescing operator, written as ?? is a logical operator that returns its right-hand side operand when its left-hand side operand is null or undefined, and otherwise returns its left-hand side operand.

Here’s how the two proposals work together to build a default value when none is found.

let customer2 = {
  name: "Paige",
  details: { 
    age: 30 // once again a city is not provided on this object
  }
};

const customerCity2 = customer2?.city ?? "City not provided";
console.log(customerCity2); // prints: "City not provided"

Since the city property is not provided and evaluates to undefined courtesy of the optional chaining operator, the, the nullish coalescing operator then kicks in and defaults to the right-hand side operand "City not provided" because the left-hand side operand evaluated to undefined.

Just think, you can create your own default values if one is missing, preventing tons of time spent debugging weird errors caused by bad data. How awesome is that? Pretty awesome in my humble opinion! 😀

Conclusion

The optional chaining operator aims to make our lives as web developers and our jobs as programmers that much simpler by eliminating errors that will break your application when object values or functions are not present.

Since the proposal is still in Stage 4 at the time of this writing it is not yet fully supported by all browsers, or standardized into the JavaScript language. Chrome, Firefox and Opera support the functionality, Edge, Safari and Internet Explorer do not.

But because it has reached Stage 4, it is now only a matter of time before it’s added to the standard ECMAScript syntax and supported across all the major browsers. You can keep an eye on the proposal in Github or the MDN Docs.

Cleaner, clearer, easier to maintain code. That’s the end game of all these improvements to the JavaScript language, and I, for one, am really excited to see the optional chaining operator coming into mainstream development soon.

Check back in a few weeks, I’ll be writing more about JavaScript, React, ES6 or something else related to web development.

Thanks for reading, I hope you’ll keep an eye out for when optional chaining gains full support and we can start simplifying our code’s null and undefined checks and putting this into production. This is going to make web development a whole lot easier.

References & Further Resources

React Socks and Responsive Web Design With React

Tue, 04 Feb 2020 00:00:00 GMT

</figure>

Introduction

A few months ago, I gave a talk with a coworker at the tech conference Connect.Tech about the ever-growing need for responsive web design when building websites and applications. We discussed a couple of different ways to approach it, specifically when it comes to the JavaScript framework React.

In case you’re unfamiliar with the term, let me quickly define responsive web design so we’re all starting at the same point. The best definition I’ve found comes straight from Smashing Magazine, a leading editorial voice for web developers and web designers.

“Responsive web design is the approach that suggests that design and development should respond to the user’s behavior and environment based on screen size, platform, and orientation.” — Smashing Magazine

It should already make sense why it’d be evermore necessary in today’s world to start with responsive design for multiple browser/device sizes in mind as you build applications, but in case you’d like a few statistics, here are some that should help to convince you.

“57% of users say they won’t recommend a business with a poorly designed mobile site.” — socPub

In other words, a poor mobile presence (or, worse still, no mobile presence at all) can make your organization look unprofessional and out of touch with the modern consumer. Here’s another:

“83% of users say a seamless experience across all devices is very important.” — Wolfgang Jaegel

Consumers today expect a seamless experience and brand consistency regardless of what device they’re using to interact with your brand. Consider this chart from August 2018-August 2019 showing the average screen sizes of users worldwide.

<figcaption>Data source: GlobalStats, showing August 2018-August 2019</figcaption>

</figure>

In 2020 and beyond, all signs point to a need for responsive design to meet users wherever they are.

While I won’t go into all the details we went into in our talk of how to approach responsive design, I’d like to share two possible approaches to responsive design for anyone else looking to get started with it.

This piece will cover the second solution for responsive web design, which is specific to the JavaScript framework React: using React Socks to conditionally render elements based on viewport size.

If you’d like to see the first solution I talked about (CSS media queries), you can see my post here.

What is React Socks?

React Socks is a React library designed to render components based on specific viewports.

While CSS media queries are used to show and hide elements always present in the DOM, React Socks goes one step further, helping to conditionally render elements based on viewport size. This means unless the components are the right size for the current viewport, React Socks won’t render the component (or its accompanying CSS) at all.

What this boils down to is:

Rendering viewport-specific components without hassle
The ability to name and define your own breakpoints where you need them (React Socks also has predefined breakpoints as well, if you prefer not to customize them)
The potential for improved app performance if you lazy load viewport-specific components
Simpler and sweeter syntax, for ease of use

Using React Socks

Implementing React Socks is actually easy, as well. To begin, you wrap your top-level component (probably App) with a <BreakpointProvider> tag, and then you can use the <Breakpoint> tag anywhere you need inside the rest of the application.

BreakpointProvider implementation

Here’s how I wrapped my app at the top level with <BreakpointProvider>.

App.js

import React from 'react';
import { BrowserRouter as Router, Route, Switch } from 'react-router-dom';
import { BreakpointProvider } from 'react-socks';
import NowPlaying from './containers/NowPlaying/NowPlaying';
import Dashboard from './containers/Dashboard/Dashboard';
import Upcoming from './containers/Upcoming/Upcoming';
import MovieSearch from './containers/MovieSearch/MovieSearch';
import Genres from './containers/Genres/Genres';
import Header from './containers/Header/Header';
import './App.css';

function App() {
  return (
    <BreakpointProvider>
      <div className="App">
        <Router>
          <LastLocationProvider>
            <Header />
            <Switch>
              <Route exact path="/" component={Dashboard} />
              <Route exact path="/search" component={MovieSearch} />
              <Route exact path="/nowplaying" component={NowPlaying} />
              <Route exact path="/upcoming" component={Upcoming} />
              <Route exact path="/genres" component={Genres} />
            </Switch>
          </LastLocationProvider>
        </Router>
      </div>
    </BreakpointProvider>
  );
}

export default App;

Import it from React Socks at the top of the file, and wrap the whole app. Done!

Once that’s taken care of, you’re free to start using the <Breakpoint> tag throughout your app wherever you need responsiveness.

Breakpoint implementation

This is an example of setting up breakpoints inside of a component using the <Breakpoint> tag.

In addition to the actual Breakpoint tag wrapper, you’ll need to include a size [React Socks’ defaults are xsmall (0px), small (576px), medium (768px), large (992px), and xlarge (1200px)] and a modifier of down, up, or only.

Down means anything up to that particular breakpoint and below it, up means anything down to that particular breakpoint and above it, and only (or no modifier) means only within that particular breakpoint’s pixel ranges. It may make more sense if you look at the example below.

The first snippet is the JSX code, and the second is the CSS that goes along with the different breakpoints.

Genres.js

    return (
      <div className="genres-page">
        <h1>Choose a Genre</h1>
        {(error || loading) && <h3 className="genre-info">{info}</h3>}
        <Breakpoint large up>
          <div className="genre-list">
            {this.renderRedirect()}
            {genreInfo}
          </div>
        </Breakpoint>
        <Breakpoint medium>
          <div className="genre-list">
            {this.renderRedirect()}
            {genreInfo}
          </div>
        </Breakpoint>
        <Breakpoint small down>
          <div className="genre-list">
            {this.renderRedirect()}
            {genreInfo}
          </div>
        </Breakpoint>
      </div>
    );

Genres.scss

.genre-list {
    display: grid;
    grid-template-columns: repeat(4, 1fr);
    max-width: 1200px;
    grid-gap: 15px;
    justify-items: center;
    margin: auto;
    h3 {
      margin: 200px auto;
    }
  }

.breakpoint__medium-only {
  .genre-list {
    grid-template-columns: repeat(3, 1fr);
  }
  h3 {
    font-size: 18px;
    width: 80%;
  }
  .genre-component {
    width: 170px;
    height: 170px;
    h3 {
      font-size: 22px;
    }
  }
}

.breakpoint__small-down {
  .genre-list {
    grid-template-columns: repeat(2, 1fr);
  }
  h3 {
    font-size: 18px;
    width: 80%;
  }
  .genre-component {
    width: 140px;
    height: 140px;
    font-size: 18px;
  }
}

In the example above, the first Breakpoint wrapper dictates that whenever a viewport is large up (i.e. 992px or greater), the first CSS listed will be rendered (the one with grid-template-columns: repeat(4, 1fr);).

The second Breakpoint wrapper will render only between the viewport sizes of 991px and 577px because its breakpoint size is set as medium. And the CSS that goes with it is conditionally rendered by being wrapped inside the class of .breakpoint__medium-only. This CSS class is added by the addition of the React Socks Breakpoint package — no modification by you is necessary.

And the third will render whenever the viewport is 576px or smaller because it’s set as small down inside the Breakpoint component. And, likewise, the CSS for including grid-template-columns: repeat(2, 1fr); will only ever render at that viewport size as well.

This kind of conditional rendering of components can shrink the CSS bundle size and clean up the CSS being loaded into the browser significantly by only loading what the view requires and nothing else.

How I used React Socks in my React app

Along with the conference talk we gave, there were three React movie demo sites I built to illustrate my point, using the Movie Database API for a data source.

The first site was completely unresponsive to show the poor user experience across multiple devices.

The second site was built with CSS media queries, making it responsive for tablet and mobile. And the third site was built using the React-specific responsive library React Socks to make it responsive.

This third demo is what I’m showing snippets of code from in this piece. If you’d like to see the whole codebase, it’s available on GitHub, and if you’d like to see a live version of the site, it’s available at this URL.

So above was one example of how to use React Socks, but let me show you another example where it really shines. When you have quite different layouts or components, you need to render based on viewport size. This is where React Socks excels.

Basic component example

Desktop and tablet views

Here are two screenshots of my movie app in both desktop and tablet size, showing the header-based navigation bar present throughout the app.

For both of the larger sizes, the links to the different pages are always visible (Now Playing, Coming Soon, Movie Search, and Genres — plus a link back to the Dashboard landing page on the far right side).

Desktop view

<figcaption>A page in my application using the desktop Header component for the navigation bar</figcaption>

</figure>

And here’s the tablet view. Once again, there’s enough room across the top for all the links to be present.

Tablet view

<figcaption>The same page in tablet view, still using the Header.js component for the navigation bar.</figcaption>

</figure>

Now, here’s a shot of the React JSX that goes along with those views.

As you can see, the Breakpoint wrapper uses medium up to display the header, while the viewport is 577px and above. When the viewport is smaller than that, React Socks makes it possible to render a completely different component named <Sidebar>.

Header.js

const Header = () => {
  return (
    <nav className="navbar-wrapper">
      <Breakpoint medium up>
        <ul className="navbar-links">
          <li className="navbar-link-logo">
            <NavLink to="/">
              <img src={TmdbIcon} alt="logo" />
            </NavLink>
          </li>
          <div className="navbar-top-links">
            <li className="navbar-link">
              <NavLink to="/nowplaying">Now Playing</NavLink>
            </li>
            <li className="navbar-link">
              <NavLink to="/upcoming">Coming Soon</NavLink>
            </li>
            <li className="navbar-link">
              <NavLink to="/search">Movie Search</NavLink>
            </li>
            <li className="navbar-link">
              <NavLink to="/genres">Genres</NavLink>
            </li>
          </div>
        </ul>
      </Breakpoint>
      <Breakpoint small down>
        <Sidebar />
      </Breakpoint>
    </nav>
  );
};

Sidebar component switched in with React Socks

Mobile size

But here is a screenshot of what the same header looks like in mobile view: It turns into a sidebar that slides in from the left with the same links available when a user taps the hamburger bar in the top-right corner.

And, once again, the link back to the homepage (the Dashboard page) is now in the right-hand corner.

<figcaption>The mobile view page in the application using the Sidebar.js component.</figcaption>

</figure>

This view is actually rendered by the Header component’s child component, Sidebar, because Sidebar is wrapped inside of the React Socks breakpoint of small down. As long as the viewport size is under 576px, the Sidebar component will render instead of the Header — with its links across the top.

Here is the JSX for the Sidebar component.

Sidebar.js

const Sidebar = () => {
  const [expandedLinks, setExpandedLinks] = useState(false);

  const toggleLinks = () => () => {
    setExpandedLinks(!expandedLinks);
  };

  return (
    <>
      <ul className="sidebar-top">
        <div className="sidebar-links">
          <li className="sidebar-link bars" onClick={toggleLinks()}>
            <i className="fa fa-bars" aria-hidden="true" />
          </li>
          <li
            className="sidebar-link-logo"
            onClick={() => setExpandedLinks(false)}
          >
            <NavLink to="/">
              <img src={TmdbIcon} alt="logo" />
            </NavLink>
          </li>
        </div>
      </ul>
        <div>
          <li className="sidebar-link" onClick={() => setExpandedLinks(false)}>
            <NavLink to="/nowplaying">Now Playing</NavLink>
          </li>
          <li className="sidebar-link" onClick={() => setExpandedLinks(false)}>
            <NavLink to="/upcoming">Coming Soon</NavLink>
          </li>
          <li className="sidebar-link" onClick={() => setExpandedLinks(false)}>
            <NavLink to="/search">Movie Search</NavLink>
          </li>
          <li className="sidebar-link" onClick={() => setExpandedLinks(false)}>
            <NavLink to="/genres">Genres</NavLink>
          </li>
        </div>
    </>
  );
};

Now, imagine trying to make those two very different components into one that could work well at both desktop and mobile sizes. It might be possible, but the JSX would be messy and the CSS even messier.

React Socks makes it so much easier to simply switch out complete components, making for cleaner code and easier development.

Pros and Cons of React Socks

I’ve covered a few different examples of how to use React Socks to switch React components in and out based on the viewport size, but I feel it’s important to talk about some of the benefits and drawbacks that come along with them.

As I alluded to above, for some situations they can be a great solution, but they may not perfect for solving every responsive design need.

Let’s talk about the highlights first.

Benefits

Ideal when you have very different components or layouts to render based on viewport size — as you’ve seen in the header/sidebar example above, the desktop and mobile views and functionality are quite different.
- Trying to shoehorn those into one component and use pure CSS media queries to show and hide different pieces of the component at different screen sizes would have been messy and more complex.
Custom breakpoints can be named and set — if you need more specific breakpoints than the default set by React Socks, it’s easy.
- If you wanted a breakpoint for extra, extra large monitors, you could simply name it giantMonitor and set a viewport size with it. Likewise, if you wanted one called pizza or bicycle, you could do that too. It’s totally up to you and your needs.
Less CSS is rendered per view than with media queries — as I mentioned before, React Socks only renders the component that’s wrapped within its Breakpoint tags (and the CSS that goes along with it).
- Smaller CSS bundle sizes can mean faster page load times, especially for mobile users, resulting in a better user experience.

React Socks only has one major drawback I identified while trying it out.

Drawbacks

The biggest drawback to React Socks: It can be overkill for code that keeps its same structure and just needs to adjust its height, width, margin, padding, etc.
- For minimal changes that keep the same layout and DOM elements, it may be quicker to just add some more CSS wrapped in a media query instead of having to define the same component multiple times wrapped in Breakpoint tags and the CSS to go with it.

And that’s really the only thing I can point out when you’re considering whether to employ React Socks or not.

Conclusion

Responsive web design is of paramount importance to every designer and developer in today’s multidevice world. We can’t even imagine some of the scenarios where applications might be used, and we can’t afford to give users a bad experience — because our competitors won’t.

There are plenty of solutions out there, but one of the approaches I chose to investigate is using the React-specific library React Socks to make sites responsive. It’s specifically tailored to work with the React framework, and it’s very easy to get started with.

React Socks renders only the components and CSS necessary for the viewport and nothing else, it’s easy to create custom breakpoints if needed, and it provides an option for cleaner, easier-to-read code and better app performance. This is a really good option for your responsive needs.

Check back in a few weeks — I’ll be writing more about JavaScript, React, ES6, or something else related to web development.

Thanks for reading. I hope you’ll consider React Socks as you’re designing and building your own responsive React-based web applications. It’s a powerful, versatile solution.

References & Further Resources

Pure CSS Media Queries and Responsive Web Design With React

Mon, 20 Jan 2020 00:00:00 GMT

import CodePen from '../../src/components/CodePen.astro';

</figure>

Introduction

Recently, I gave a talk with a coworker at the tech conference Connect.Tech about the ever-increasing need for responsive web design when building applications, and then a couple of different ways to approach it, specifically when it comes to the JavaScript framework, React.

In case you’re not familiar with the term, let me quickly define responsive web design so we’re all starting at the same place. The best definition I’ve found comes straight from Smashing Magazine, a leading editorial voice for web developers and web designers.

“Responsive web design is the approach that suggests that design and development should respond to the user’s behavior and environment based on screen size, platform, and orientation." — Smashing Magazine

It should already make sense why this would be more and more necessary in today’s world to start with responsive design for multiple browser/device sizes in mind as you build applications, but in case you’d like a few statistics, here are some that should help to convince you.

“In 2018, 52.2% of all website traffic worldwide was generated through mobile phones, up from 50.3% in the previous year." — Statista

And:

"On average, 88% of global consumers report they multiscreen, using 2.42 devices at the same time." — Adobe

Or consider this chart from August 2018-August 2019 showing the average screen sizes of users worldwide.

<figcaption>Data source: GlobalStats August 2018 — August 2019.</figcaption>

</figure>

In 2020 and beyond, all signs point to a need for responsive design to meet users where ever they are.

While I won’t go into all the details in our talk of how to approach responsive design (if you’d like, you can see the full slide deck of the talk), I would like to share two possible approaches to responsive design for anyone else looking to get started with it.

This article will cover the first solution for responsive web design, which would really work for any JavaScript application as long as it’s using CSS for its styling: CSS media queries.

What are CSS media queries?

Media queries are a popular technique for delivering a tailored style sheet (responsive web design) to desktops, laptops, tablets, and mobile phones.

CSS media queries are usually used to check things like:

Width and height of the viewport.
Width and height of the device.
Orientation (is the device in landscape or portrait mode?).
Resolution.

They can also be used to specify that certain styles are only for printed documents or for screen readers.

Anatomy of a CSS media query

But by far the most common uses for media queries fall into the first set of bullet points above: checking viewport and device widths and heights to know which CSS styling to serve up.

And here’s an example of what the CSS media query code will look like.

Typical example of a CSS media query

@media only screen and (min-width: 375px) and (max-width: 969px) {  
  body { 
    background: green;
    font-size: 16px;
  }
}

The @media rule, introduced with CSS3, is used to apply different styles for different media types/devices. only screen applies to all computer screens, tablets, smartphones, etc.

In the example above, this start specifies that the CSS styles ( background: green and font-size: 16px ) defined within will only apply to the body HTML tag for screens between the sizes of at least 375px (min-width: 375px), all the way up to screens of 969px (max-width: 969px).

This is a very basic example, but CSS media queries are incredibly powerful. They can be used to show and hide different elements of the DOM, resize fonts, and switch around whole page layouts with relative ease.

Before I show you how I used them though, let’s talk about the two basic design approaches to take with media queries.

Approaches to CSS media queries

There are two common approaches to writing CSS media queries. They’re known as subtractive CSS and additive CSS.

Neither one is really better than the other, it depends more on the main audience of the application to help determine which approach is right for you.

Questions to help figure this out might include: "Will it be used more on mobile devices or desktops, or will it be a large group of diverse users, or a small internal group of users within a company?" Things like that.

Here are more details about the two approaches.

Subtractive CSS

The first approach, subtractive CSS, is the one that’s probably more commonly taken with applications that will have smaller, more homogenous user bases.

Think internal applications used within organizations for very specific purposes, that oftentimes can only be accessed from a specific device — either because of company firewalls and networks or hardwired terminals, etc.

Subtractive CSS generally starts with the comps and screen styles at the largest resolution first. It begins with the largest displays, and then media queries are added to undo unneeded CSS for the smaller viewports.

Basically, it puts in a bunch of styling that undoes some of the more complex layouts that have been laid out at a higher resolution.

Even though it has subtractive in the name, it usually results in more lines of CSS having to override underlying styles.

Here’s a CodePen you can look at, illustrating subtractive CSS at work. Check out the amount of CSS I had to add to undo the layout from a widescreen view to accommodate a smaller viewport.

If you expand and shrink the viewport, you can watch the styles change for large screen, tablet, and mobile.

Subtractive CSS demo

Additive CSS

Additive CSS is the opposite of subtractive, as the name implies. With additive CSS, you start with the smallest resolution styling first and add media queries to build up the CSS as the screen size increases.

It begins simply, and gets more complicated with the layout as horizontal screen real estate increases.

Additive CSS is probably the more recommended way to begin designing a new application in general. It will make the amount of CSS being loaded at different breakpoints smaller if it's designed mobile-first and has less CSS to override and undo (better for your end-user experience).

Also, there’s a very good chance users will be accessing the application from a whole variety of different screen sizes and devices you can’t even imagine, and having contingency categories they’ll fall into is vital.

Here’s another CodePen showing additive CSS at work. You can see that in this example, I start with the smallest viewport first, and slowly add more details and layout complexity as the viewport size increases under media queries.

Try playing with the viewport size to see the CSS changes.

Additive CSS demo

I know that both of these examples may seem rather simplistic, but just imagine a larger, multi-screen application (heck, look at most of the apps you use on your phone and your laptop on a regular basis), and think about how you might build them using either approach.

How I used CSS media queries with React

Along with the conference talk we gave, there were three React movie demo sites I built to illustrate my point, using the Movie Database API for a data source.

The first site was completely unresponsive to show the poor user experience across multiple devices.

The second site was built with CSS media queries making it responsive for tablet and mobile.

And the third site was built using a React-specific responsive library called React-socks, to make it responsive.

This second demo is what I’ll be showing snippets of code from in this blog. If you’d like to see the whole codebase, it’s available on GitHub, and if you’d like to see a live version of the site, it’s available here.

Basic component example

Desktop view

I’ll be showing you a portion of the CSS for the MovieDetails React component in my application. Here’s what the desktop page looks like that uses this component.

</figure>

As you can see from the screenshot, at desktop size the layout is pretty complex: movie poster backdrop, the official trailer poster on the left side, the movie synopsis, release date, ratings and reviews on the right side, etc.

And here is a snippet of the CSS involved to make the layout for the movie poster happen. This was the base CSS I used to build my smaller screen views from. Yes, this is an example of subtractive CSS.

MovieDetails.scss

.movie-details-poster-wrapper {
  display: grid;
  grid-template-columns: 1fr 1fr;
  grid-template-rows: 3% 450px 450px 80px;
  grid-gap: 15px;
  justify-items: center;

  .movie-details-poster {
    border-radius: 10px;
    border: solid #847f86 10px;
    grid-row: 2;
  }
}

From the code, you see I’m using a CSS Grid layout here for the desktop layout. Now, let’s see how the view and component’s CSS changes as the screen size shrinks.

Same component with media queries

Tablet view

Here is the look of that same React component MovieDetails at tablet size.

</figure>

At tablet viewport size, the movie poster gets centered in the viewport and the synopsis, ratings, and reviews slide underneath it.

Here’s what a portion of the CSS looks like that goes with this view.

MovieDetails.scss

@media only screen and (min-width: 550px) and (max-width: 1089px) {
    .movie-details-poster-wrapper {
      display: flex;
      flex-direction: column;
      align-items: center;
    
    .movie-details-poster {
        margin: 20px 0;
      }
    }
  }

For the tablet view, the CSS layout switches from Grid, which is a two-dimensional way of designing layouts to CSS Flexbox, which is a more one-dimensional way of arranging layouts. More appropriate for a view where each element is just laid out in order, one after another.

Straightforward enough.

On to the last example: the mobile view.

Mobile view

</figure>

The mobile view more closely resembles the tablet view for the MovieDetails React component. Once again, the movie poster is front and center, the overview and ratings come after.

For mobile, the font size has also increased and now there’s a toggle on the reviews to "Read more..." instead of the user being able to scroll through the whole review from the start as they can at larger view sizes.

Here’s the last example of the CSS that accompanies it.

MovieDetails.scss

@media only screen and (max-width: 549px) {
    .movie-details-poster-wrapper {
      display: flex;
      flex-direction: column;
      align-items: center;

    .movie-details-poster {
        width: 300px;
        margin: 20px 0;
        overflow-y: scroll;
      }
    }
  }

As you can see, the .movie-details-poster-wrapper's display property stays in CSS Flexbox at this viewport size, and its flex-direction remains column so that all the child elements contained within line up nicely.

You’ll notice though that for this view, the .movie-details-poster has a width property set to keep it from going beyond the width of the viewport (the actual image supplied from TMDB’s API is 500px large, by default), along with its other styling properties.

Makes sense? Good, now I’ll give a quick overview of the pros and cons I encountered while using CSS media queries.

Pros and Cons of CSS media queries

I’ve covered the basics of CSS media queries and how you’d use them with a component in React, but I feel it’s important to talk about some of the benefits and drawbacks that come along with them.

Sometimes they can be a great solution, but they may not perfect for solving every responsive design problem.

Let’s start with the highlights.

Benefits of CSS media queries

Great when reusing the same components just slightly in different layouts — as long as the location or visibility of elements within a component doesn’t differ too much, CSS media queries should be able to do the job. When completely new components need to be added at different viewport sizes, the JavaScript and JSX component code can start to get complicated.
Breakpoints can be reused across components — this is a nice time-saver. Oftentimes, the breakpoints for viewports you’ll set for one component will also make sense for other components, so there’s less need to reinvent the wheel for each page of an application.
Separation of concerns — for people who aren’t in the camp of CSS-in-JS (and I know there’s plenty), CSS media queries provide a nice separation of logic and presentation. The CSS strictly handles the layout, and the JavaScript and React framework handles the data fetching and logic manipulation.

But while there are a lot of good points about CSS media queries, there are some shortcomings.

Drawbacks of CSS media queries

CSS can get long and messy quickly — if you’ve got three or more breakpoint sizes for one component and you’re using more of a subtractive style of CSS, your CSS file can get long.

For instance, the MovieDetails.scss file I referenced for my examples above, is 176 lines long before being bundled up by webpack for deployment. And that’s a relatively less-complex example, it can get much more complicated than that.

Very different layouts from one size to another requires duplicated code with display: none to achieve results.

This one is pretty self-explanatory, but if you’ve got a desktop layout with a title element at the top then an image underneath, but the mobile layout has an image above the title, there’s probably no way to write the JSX so you don’t end up duplicating title in two places in the code. This means duplicate elements in one component at different viewport levels — which means less readability and more complexity in maintaining your code.

CSS media queries are great, but they’re not always the right answer and it’s important for you to know that before employing them.

Conclusion

Responsive web design is something every designer and developer needs to be thinking of today. We can’t even imagine some of the scenarios where applications might be used, and we can’t afford to give users a bad experience, because our competitors won’t.

There are plenty of solutions out there, but one of the approaches I chose to investigate is using CSS media queries within a React application. They’re native to CSS3, which means, regardless of your JavaScript framework, as long as it uses CSS for the styling, you can leverage them.

To implement them requires very little extra CSS, at least initially. And if you structure the CSS right (with an additive CSS approach that builds up as the screens get larger), the mobile CSS bundle size can be pretty small meaning a faster load time for users and a better experience. It’s a pretty solid option.

Check back in a few weeks, I’ll be writing the second piece of this series on responsive design with React.js. This piece will focus on a React-specific solution for making sites responsive.

Thanks for reading, I hope you’ll give CSS media queries consideration as you’re designing and building your own responsive web applications. They’re a powerful, versatile solution.

References & Further Resources

JavaScript Error Handling from Express.js to React

Mon, 23 Dec 2019 00:00:00 GMT

</figure>

Introduction

I've used a full stack MERN (MySQL, Express, React and Node.js) project I made from scratch for a whole handful of blog posts now. As I slowly built it out and added more bells and whistles to it, I learned a slew of new things worth sharing to help other devs avoid obstacles that I ran into along the way.

Past topics included using Sequelize as the ORM for a MySQL database, using PassportJS and JWT for authentication and protected routes, using Nodemailer to provide password reset capability via email, etc. Honestly, working through building this application, just for the sake of learning more Node, more React, more core JavaScript, taught me so much.

If you want to see the rest of my posts pertaining to this repo and the code itself, I’ll link to them all at the bottom of this article.

And today’s post is no exception.

Today, I’m talking about how to properly handle errors on both sides of a JavaScript application: from throwing errors on the server side in Express, to catching and handling them on the client side with React and Axios.

Express error handling

Since the server side is where the majority of the errors occur from, starting with the Node.js / Express server code is where I’ll begin as well.

I must confess, for quite a while, in the course of building this app, I passed back HTTP 200 status (the OK status) codes from the Express server even when an error like a missing JWT token or user ID was not found in the database.

Improper Express error handling example

res.status(200).json("404 - no user exists in db to update");

I did this because I wanted to send not just a 401 (unauthorized) or 403 (forbidden) HTTP status code, but because I also wanted to send a written message of what exactly the error was on the server side, and for the longest time, I couldn’t find out how to do both with Express.

It was either send an error message through res.json("This is an error message the client can read") with a 200 HTTP status, or send a 401 or 403 HTTP status with no further info on what actually went wrong with a res.status(401).

Then one day, finally, I found the solution I was looking for buried deep in the Express documentation, of which there is quite a lot. And it really wasn’t as tricky as I thought — it was similar to how I sent a message along with a 200 HTTP status when something went right, like so:

Express success status code message example

res.status(200).send({ message: "user created" });

I could do the same with an error status and message.

Express failure status code and message example

res.status(401).send("no user exists in db to update");

Just by switching from Express’s .json() to .send() method meant I could chain together an HTTP error code and informational message for the client to decipher and then handle appropriately on the browser side. 😄

It is worth noting that .send() can handle objects, strings, or arrays in JavaScript. The success message example is in the shape of an object with the property of message while the failure response is just a string because in the event of a successful response things like a message, a JSON Web Token or other data might be sent over to the client from the server. My errors, however, needed nothing more than an HTTP response and error message as a string. Just so you know for the future. 🙂

Below is an example of one of the server side routes in my application showing the full implementation of sending either a success or failure when a user tries to update their personal info while logged in. (I’ve chosen to include it as both the nicely formatted and highlighted image from VS Code and a copy-able code snippet).

updateUser.js

module.exports = (app) => {
  app.put("/updateUser", (req, res, next) => {
    passport.authenticate("jwt", { session: false }, (err, user, info) => {
      if (err) {
        console.error(err);
      }
      if (info !== undefined) {
        console.error(info.message);
        res.status(403).send(info.message);
      } else {
        User.findOne({
          where: {
            username: req.body.username,
          },
        }).then((userInfo) => {
          if (userInfo != null) {
            console.log("user found in db");
            userInfo
              .update({
                first_name: req.body.first_name,
                last_name: req.body.last_name,
                email: req.body.email,
             })
            .then(() => {
              console.log("user updated");
              res.status(200).send({ auth: true, message: "user updated" });
            });
         } else {
          console.error("no user exists in db to update");
          res.status(401).send("no user exists in db to update");
         }
       });
      }
    })(req, res, next);
  });
};

Let me quickly break down what’s happening in this file.

The first thing that happens when the route is hit is that PassportJS takes over and checks to see if the user’s JSON Web Token both exists and is valid, this happens in another file and is covered in a different blog post so I won’t go in depth about that now.

Once a user’s been authenticated, the route goes through the process of attempting to find that username in the MySQL database, and if it is found, the user info is updated and a success message with HTTP status 200 and a message of "user updated" is sent back.

If the username is not found in the database the server logs that the error that the user doesn’t exist and sends back a 401 HTTP status and message that "no user exists in db to update".

Here’s exactly what gets logged on the server side console that you’d see looking through the logs.

Error Express throws in the server logs

The main piece of code to focus on in terms of the error handling is this block right here towards the bottom of the file.

<figcaption>The error printed out to the server’s console and sent to the client side.</figcaption>

</figure>

That’s about all there is to successfully sending both bad status codes and messages from Express to the browser. Now let’s talk about how the browser handles those errors.

{
  console.error("no user exists in db to update");
  res.status(401).send("no user exists in db to update");
}

React & Axios error handling

For the client side of my full stack application I chose to use the extremely popular promise based HTTP library Axios instead of the browser’s native Fetch API, as I find it easier to work with. This example will be shown using Axios for all HTTP requests and error handling.

Just as I was slowly improving the error handling on the server side, I was also improving my error handling on the client side as my application progressed and improved.

Before I properly handled the errors the server threw, this was all I was doing to interpret errors.

Poorly handling errors thrown from the server to the client example

  .catch(error => {        
    console.log(error.data);             
    this.setState({         
      loadingUser: false,         
      error: true,       
    });
  });

I would attempt to catch an error, log its message out to the console and then set the React application’s state accordingly. Not terrible for a start, but also not great for a final solution — the second I would try to throw something outside of a 200 HTTP status with a message, the client would just blow up in the catch() block.

And just like with the server side, I finally got tired of this half baked error handling I was doing and found some better Axios documentation around how the library wants you to handle errors, which I’d overlooked before.

Turns out the way to read and handle errors with Axios is actually error.response.data instead of just error.data in the catch() function. It’s not well highlighted but the documentation notes:

 catch(error => {
  if(error.response) { 
    /* the request was made and the server responded
    with a status code that falls out of the range of 2xx */
    console.log(error.response.data)
  }
}

And with that one revelation, suddenly the error catching on the client side suddenly became much easier and more descriptive.

The way Axios interprets successful responses from the server is almost exactly the same way it interprets error codes.

Axios success response handling

try {      
  const response = await axios.get("api/findUser", {
    params: {          
      username,        
    },        
    headers: { Authorization: `JWT ${accessString}` },      
  });      
  console.log(response.data); 
  // this.setState and so on after response is received...
}

Axios failure response handling

catch (error) {      
  console.log(error.response.data);      
  this.setState({        
    loadingUser: false,        
    error: true,      
  });    
}

Which turned out to be much easier for me to update all my catch() blocks to accept this new error format than I expected it to be.

Once more, here is a large snippet from the file where a user updates their data from the client side and sends it to be saved into the database.

UpdateProfile.js

updateUser = async (e) => {
  const accessString = localStorage.getItem("JWT");
  if (accessString === null) {
    this.setState({
      loadingUser: false,
      error: true,
    });
  }
  const {
    first_name, last_name, email, username
  } = this.state;
  e.preventDefault();
  try {
    const response = await axios.put(
      "http://localhost:3003/updateUser",
      {
        first_name,
        last_name,
        email,
        username,
      },
      {
        headers: { Authorization: `JWT ${accessString}` },
      },
    );
    // eslint-disable-next-line no-unused-vars
    console.log(response.data);
    this.setState({
      updated: true,
      error: false,
    });
  } catch (error) {
    console.log(error.response.data);
    this.setState({
      loadingUser: false,
      error: true,
    });
  }
};

Here’s what’s happening in this function call.

First, updateUser() pulls the JWT token from local storage, where it was previously added.

If the token is found, Axios constructs the body of the PUT request which contains all the user’s new information, and sets the header to the JWT token the server requires for authentication.

Finally, the response comes back as either successfully updated or as a failure, in which case the error.response.data is logged to the console to help the developer figure out what went wrong and the app’s state is set to error to help the user understand what happened as well.

This is what a user would see if the dev tools console was open when the failing request was made.

Error React throws in the console logs

<figcaption>The error a developer would see in the developer tools console in the browser.</figcaption>

</figure>

The client sees the 401 status code the user sent back as well as the message from the server saying no user was found in the database with the matching user name.

The code that makes that happen is this snippet in particular.

} catch (error) {
      console.log(error.response.data);
      this.setState({
        loadingUser: false,
        error: true,
      });
    }

This wraps up my explanation of how the client can handle and display errors thrown from the server. Once again, not too tough to grasp, but just putting it all together can sometimes prove the trickiest part of all.

Conclusion

Errors in code are like death and taxes, they’re bound to happen. Our job as developers is to make sure we know when they happen and handle them gracefully so the user can have a good experience with our applications.

Although this type of thing is common place and of paramount importance, sometimes the documentation around such error handling is buried deep or is slightly cryptic with the bare bones code snippets provided. Which is why I wanted to write today detailing exactly how I pass error codes and messages from Express all the way to the Axios library being used on the React front end.

Check back in a few weeks, I’ll be writing more about JavaScript, React, ES6 or something else related to web development.

Thanks for reading, I hope I’ve given you some good solutions of how to handle errors in your own full stack JavaScript applications in the future. Errors will occur — let’s handle them well.

References & Further Resources

How To Pass Multiple Route Parameters in a React URL Path

Mon, 09 Dec 2019 00:00:00 GMT

</figure>

Introduction

Recently, I came across an interesting problem I hadn’t encountered before. I wanted to pass more than one route parameter in a React URL route path, then access those parameters in a component upon redirect.

The reason for this problem arose when I was building a demo project to illustrate various ways to do responsive web design with React. I was using The Movie Database API to supply movie data to my React front end project.

What I wanted was for a user to see a list of movie genres (action, adventure, comedy, etc.), click on a genre, and be redirected to a list of movies that fell into that genre. Seems straightforward enough, right?

<figcaption>This is what a user should see after clicking on "Action" in the genres list: movies that are classified as "adventure" movies.</figcaption>

</figure>

Well, yes, it seems simple...except that the movie API supplying the list of movies based on genre required a genreId to fetch the data, while what I wanted to display to users along with the movie list was the genreName that they’d actually clicked on to generate this list. Do you see my problem?

Two different pieces of data, both required by a different, unrelated component upon redirect (so passing data through props like with a typical child component was out of the question) from the generic Genres component to the specific GenreList component displaying the movies related to that genre.

Surprisingly, when I started to look for examples of how to pass multiple route parameters in a URL path in React, I couldn’t find much.

So that’s what I wanted to share today: How to add, and later access, multiple route parameters in a ReactJS project upon redirect using react-router-dom, the most popular router for React applications.

React routes and route parameters

Before I go much further, let me clarify what routes and route parameters are in ReactJS, in case you’re not familiar.

I’ll be using my demo project repo, so if you’d like to see the complete codebase or the working demo, they’re available here and here.

If you’re completely new to how routing works in ReactJS, I recommend checking out react-router-dom’s documentation first. This will not be an in-depth explanation of that, but it is the router I’m using in this article.

Routes

Routes specify the path to different components in a ReactJS application. When a user clicks a link to a page or hits a button that redirects him/her, the router renders the matching <Route/> for that component. For instance, this is what the route for the Genres page in my movie app looks like.

Traditional, static route URL in React:

<Route exact path="/genres" component={Genres} />

This is a static route that will always be the same, regardless of who the user is or which page they’re coming from. No matter what, whenever they click a link to go to the Genres page, this route will always be used to get them there, and it will always render the same component — no dynamic content required.

Route parameters

Route params are parameters whose values are set dynamically in a page’s URL. This allows a route to render the same component while passing that component the dynamic portion of the URL, so that it can change its data based on the parameter.

Examples of URL route params you might recognize include things like IDs for products, books, users, or, in my case, movies. At least with React, in order to get the details for the correct item, the dynamic component would need the unique, specific ID of the item supplied, so it could get the correct data to display.

Here’s what the MovieDetailsContainer route with dynamic URL parameters in my app looks like.

Dynamic route URL in React:

<Route exact path="/movie/:id" component={MovieDetailsContainer} />

This is an example of a dynamically updating route. The first part of the path, the "/movie/” part, doesn’t change, but the :id is a route parameter that is set dynamically — usually from a user’s actions, like clicking a link or typing an input and clicking a search button.

This is what changes and it causes the MovieDetailsContainer to be rendered with the ID of the movie included. The component then pulls this out of the URL path when it renders to get the details of that particular movie to show to a user.

Accessing dynamic route params in a component

Typically, the component being rendered on redirect will access the dynamic route info it needs from the URL.

Accessing dynamic path parameter in a URL:

this.props.match.params.id

Inside a React component that needs URL path data, this is the code you’ll have to use to access the route’s actual ID, which was shown as a placeholder in the route path as :id. It’s a little verbose, I agree, but up until react-router-dom’s React hooks updates, it was what we had to use to access the data.

From there it can be set to local component state, used to fetch data, rendered out in the JSX, etc.

That is pretty much where the online tutorials stop: with a redirect example showing a single :id path param. This is where my piece picks up.

Setting up the new dynamic route path

Hopefully, we’re on the same page now. Routes are clear. Route params are clear. Accessing those parameters from a component that a user is redirected to is clear. The only thing that still remains unclear is what to do if you have more than one dynamic route parameter to add to a route path.

Knowing what I knew about React route paths in react-router-dom, I decided to try creating a route path for the GenreList component. This is the component that shows the list of movies associated with a particular genre using two dynamic parameters: :genreName and :genreId.

NOTE: All of my actual routing for the app was contained in the App.js file inside the src/ folder, for clarity.

App.js

<Route
  exact
  path="/genres/:genreName/:genreId"
  component={GenreList}
/>

This was step one of handling a route containing more than one route parameter. The next step is setting the genre ID and genre name in the Genres component.

Adding more than one route parameter to a URL

The Genres component contains, among other things, two state objects: selectedGenre (a number initially set to 0) and selectedGenreName (a string initially set to ""). These are the two pieces of state which will be set by a user clicking one of the genres listed in the list.

As soon as a user clicks a single genre, the genre’s ID and name are set in the Genres component’s state. This triggers the renderRedirect() function you see below, to redirect the user to the GenresList component, as defined by the route path.

this.state.selectedGenreName is inserted as the route placeholder :genreName. this.state.selectedGenre is replaced with the route placeholder :genreId.

Genres.js

renderRedirect = () => {
  if (this.state.selectedGenre !== 0 && this.state.selectedGenreName !== '') {
    return (
      <Redirect 
        to={`/genres/${this.state.selectedGenreName}/${this.state.selectedGenre}`}
      />
    );
  }
};

Step two is now complete. A particular genre ID, required by the unrelated GenreList component to fetch movies of that genre is part of the URL, and the genre name, to display which genre the user’s viewing movies for, is also included. Let’s go to the redirect component, GenreList, that now needs this data.

Accessing the route parameters from a URL

OK, last step: the component being redirected to, GenreList, needs to pull out the different pieces of the dynamic React route path. It needs the genreId to fetch the movie list from The Movie Database API, which it extracts with this.props.match.params.genreId, and it needs genreName to display in the component’s JSX , showing which genre the user clicked on, which is accessed via this.props.match.params.genreName.

Pretty cool!

GenreList.js

if (this.props.match.params) {
  try {
    const movies = await movieAPI.getMoviesByGenre(
      this.props.match.params.genreId,
    );
    this.setState({ movies, loading: false });
  } catch (err) {
      this.setState({ loading: false, error: true });
    }
  }
}
render() {
  return (
    <>
      <div className="genre-search-title" onClick={() => this.props.history.push('/genres')} >
        <div>
          <i className="fa fa-chevron-left" aria-hidden="true" />
          <p>Back to Genres</p>
        </div>    
        <h1>{this.props.match.params.genreName} Movies</h1>
      </div>
      <MovieList
       loading={this.state.loading}
       error={this.state.error}
       movies={this.state.movies}
      />
    </>
  );
}

I didn’t believe it could be that simple for a redirect, but it was!

Just by using the correct URL path params syntax (this.props.match.params...) and passing the corresponding variable name (genreId or genreName), I was able to pull out the correct value.

This sort of thing can be done anywhere you need dynamic data when redirecting from one unrelated component to another.

Conclusion

Dynamic redirects are a fairly common occurrence in any web application, React apps included, but despite this, tutorials illustrating them are not so easy to come by. And examples of redirects with more than one piece of dynamic data are non-existent, which is why I decided to write a piece about just that.

With the help of react-router-dom's route paths and dynamic parameters, it really isn’t as complicated as I once thought to pass more than one piece of data from one component to another unrelated component via a redirect.

Check back in a few weeks, I’ll be writing more about JavaScript, React, ES6 or something else related to web development.

Thanks for reading, I hope I’ve made ReactJS redirects with dynamic URL route params a little clearer and that this was something you’ll find useful in your own applications.

References & Further Resources

Destructuring Arrays & Objects: JavaScript ES6 Feature Series (Pt 10)

Sun, 24 Nov 2019 00:00:00 GMT

</figure>

Introduction

The inspiration for these pieces is simple: JavaScript, and especially some of the new ES6+ releases, can be more than a little perplexing at first glance to many developers. To put it bluntly: what they once thought they understood becomes a completely new beast when another layer of syntactic sugar, designed to make our lives easier, is added.

That being said, according to Wikipedia, JavaScript still powers almost 95% of the 10 million most popular webpages today.

JavaScript only continues to play an evermore crucial part in the web, and I wanted to provide a bunch of articles and examples of ES6+ features that I use regularly, for other developers to reference.

The aim is for these pieces to be short, in-depth explanations of various improvements to the language that I hope will inspire you to write some really cool stuff using JS. Who knows, you might even learn something new along the way. 😄

For my final installment in this series, I will tackle array and object destructuring: the most concise way to pull out values and properties into individual variables with very little effort.

Destructuring sounds so simple at first...

The destructuring assignment, first introduced with ES 2015, is one of my favorite additions to the JavaScript standard syntax. As I said above, destructuring makes it possible to unpack values from arrays, or properties from objects, into distinct variables.

This may sound simple when you first hear it, but actually putting it into effect, especially with deeply nested arrays or objects, is a little trickier to grasp. But before I get into the parts that might trip you up, let’s just talk about what destructuring looks like in both arrays, and more recently, objects.

Array destructuring

Array destructuring was introduced and finalized into the ECMAScript language before object destructuring.

And just as object and array literal expressions provide an easy way to create packages of data, at will, like so:

const a = ["alpha","beta", "gamma", "delta", "epsilon"];

The destructuring assignment of those same arrays uses similar syntax, but on the left-hand side of the assignment to define what values to unpack from the sourced variable.

Anatomy of array destructuring syntax

let b, c, more;

const a = ["alpha","beta", "gamma", "delta", "epsilon"];
[b, c] = a;

console.log(b); // "alpha"
console.log(c); // "beta"

[b, c, ...more] = ["alpha","beta", "gamma", "delta", "epsilon"];

console.log(more); // [ "gamma", "delta", "epsilon" ]

The short example above demonstrates array destructuring and using the rest pattern to assign extra values, which I wrote about here, on the array named a.

Notice that the variables b and c are wrapped in brackets ([b, c]), and are assigned to equal the a array. [b, c] = a;

When those values are called they will take the first and second elements in the array. console.log(b); // "alpha" and console.log(c); // "beta";.

The rest pattern comes in with the variable more. That variable is designed to inherit all the rest of the values in the a array by putting the rest syntax (...more) ahead of the variable name when adding it to the array. Then, when more is called, it prints out the rest of the values in its own array. console.log(more); // [ "gamma", "delta", "epsilon" ].

Alright, let’s go through some different uses for array destructuring.

Basic variable assignment

The most basic type of array destructuring is taking one array of values and assigning those values to an equal amount of new variables.

Example of basic array variable destructuring assignment

const govtBranches = [ "executive", "judicial", "legislative" ];

const [branch1, branch2, branch3] = govtBranches;

console.log(branch1); // "executive"
console.log(branch2); // "judicial"
console.log(branch3); // "legislative"

Each of the three values of govtBranches has a corresponding variable wrapped in the array brackets, branch1, branch2, or branch3 that it is assigned to.

Then, if any of those variables are called, they reflect one of the individual values from the govtBranches array.

Assignment separation from declaration

Next up is when a variable can be declared first (either using the keywords let or var, not const), and then have its value assigned via destructuring.

Example of variable declaration and then value assignment with destructuring

let agency1, agency2;

[agency1, agency2] = [ "FBI", "CIA" ];

console.log(agency1); // "FBI"
console.log(agency2); // "CIA"

As you can see, the variables agency1 and agency2 are declared as undefined variables first. Next, they’re wrapped in array brackets and assigned to the array containing the values "FBI" and "CIA". From there, each variable can be called individually, and it represents one of the values in that array.

Default values

Interestingly, you can assign variables a default value, similar to default function parameter values, in case a value unpacked from an array turns out to be undefined.

This kind of thing may happen when there’s more variables created than values in the array it’s destructuring.

Example of default values in array destructuring

let one, two;

[one="a", two="b"] = ["c"];

console.log(one); // "c"
console.log(two); // "b"

The two variables one and two are assigned the default values "a" and "b" before being assigned to the array containing the single value of "c".

And when the variable one is called after the array destructuring takes place, its value is overwritten to be "c" due to the value in the array. The value of two does not change though because the array doesn’t contain a second value, so that value, if it existed, would be undefined.

Swapping variables

Did you know that two variable values can be swapped in one destructuring assignment? They can, and it’s pretty handy that you don’t need a temporary variable to make this possible anymore.

Example swapping variable values with array destructuring

let boots = "cat";
let rocky = "dog";

[boots, rocky] = [rocky, boots];

console.log(boots); // "dog"
console.log(rocky); // "cat"

Initially, the variable boots is a "cat" and the variable rocky is a "dog", but simply by switching the order of the values in the array they’re being assigned to, their values can be swapped so boots becomes the "dog" and rocky becomes the "cat".

This is a really useful trick in certain scenarios.

Parsing an array returned from a function

Getting an array returned from a function is nothing new, but now you can destructure the values being returned to make working with them more concise.

Example parsing an array returned from a function

function color() {
  return ["red", "yellow", "green", "blue"]
}

let r, y, g;

[r, y, g] = color();

console.log(r); // "red"
console.log(g); // "green"

The color() function returns an array of colors. By destructuring the variables r, y, g against the function, those values in the array are assigned to those variables.

Ignoring some returned values

In the same vein, destructuring lets you ignore certain array values you aren’t interested in.

Example ignoring values from an array with destructuring

function ignoreColor() {
  return ["indigo", "orange", "lime"]
}

const [i, ,l] = ignoreColor();

console.log(i); // "indigo"
console.log(l); // "lime"

Simply by adding an empty space in the destructured array, you can choose not to return the value "orange" from the function ignoreColor().

You can also choose to ignore all the values from a function if you’d like (though I don’t really see much of a use case for that).

[ , , ] = ignoreColor();

Assigning the rest of an array to a variable

And I’m back again to part of what I showed in the very first array destructuring example: using the rest operator (...) to pick up any leftover values from an array.

Example assigning leftover values to a variable via array destructuring

const [commanderInChief, ...staff] = ["President", "Vice President", "Chief of Staff", "Press Secretary"];

console.log(commanderInChief); // "President"
console.log(staff); // [ "Vice President", "Chief of Staff", "Press Secretary" ]

Just as before, commanderInChief takes the first value in the array, and by using the rest syntax, ...staff takes all the remaining values in the array as a new array of its own.

Simple as that. Now let’s take a look at how destructuring works on objects.

Object destructuring

Object destructuring takes a similar tack as array destructuring, except instead of values being pulled out of an array, properties (keys) and their values can be pulled out of an object.

Here’s some examples of how that can look.

Basic object destructuring assignment

Once again, I’ll start with the most basic example of how object destructuring.

Example of basic object destructuring

const pieIngredients = { pumpkin: "1 can", pieCrust: "1 crust", spice: "2 tsp"};

const { pumpkin, pieCrust, spice} = pieIngredients;

console.log(pumpkin); // "1 can"
console.log(pieCrust); // "1 crust"
console.log(spice); // "2 tsp"

By wrapping the properties in the object pieIngredients and setting them equal to the object, each property, pumpkin, pieCrust, and spice, becomes its own variable and the value attached to it becomes the value for the new variable.

Assignment without declaration

A variable can also be assigned its value with destructuring separate from its declaration, just like you can do it with array destructuring.

Example of assigning variables after declaring the variables separately

let hobby, sports;

({hobby, sports} = {hobby: "knitting", sports: "croquet"});

console.log(hobby); // "knitting"
console.log(sports); // "croquet"

Note the parentheses ( ... ) around the assignment statement are required when using object literal destructuring assignment without a declaration.

Otherwise, the syntax is considered invalid because the syntax on the lefthand side, the {hobby, sports}, is considered a block and not an object literal. Putting the parentheses around the whole line though, clarifies the intent and makes it valid.

Assigning to new variable names

One helpful thing is that a property can be unpacked from an object and assigned to a variable with a different name than the object property.

Example reassigning destructured object properties to new variable names

const car = {speed: 110, color: "red"};

const { speed: fast, color: cherry } = car;

console.log(fast); // 110
console.log(cherry); // "red"

Here, for example, const {speed: fast} = car; takes from the object car the property named speed and assigns it to a local variable named fast.

Default values

Just like with array destructuring, variables for destructured objects can be assigned a default, in the case that the value unpacked from the object is undefined.

Example of assigning default values to object destructuring variables

const { redWine = "cabernet", whiteWine = "pinot grigio"} = { redWine: "malbec"};

console.log(redWine); // "malbec"
console.log(whiteWine); // "pinot grigio"

In this example, the variables redWine and whiteWine are assigned default values of "cabernet" and "pinot grigio". Then the redWine variable is reassigned the value of "malbec", but since whiteWine is not defined in the object being destructured, it retains its original value.

Unpacking fields from objects passed as function parameters

Another feature of object destructuring is that you can actually use destructuring syntax inside of function calls to get just those values back. Take a look at this.

Example of passing destructured object properties as function parameters

const girl = {
  name: "Paige",
  age: 30,
  eyeColor: "blue",
  hair: {
    type: "curly",
    color: "red",
    length: "shoulder-length"
  }
}

const getUserName = ({name}) => {
  return {name};
}

console.log(getUserName(girl)); // { name: "Paige" }

const getUserHair = ({hair: {type, color}}) => {
  return `Her hair is ${color} and ${type}`;
}

console.log(getUserHair(girl)); // Her hair is red and curly

In the example here, the object girl is a pretty standard one. It’s got two levels of nested properties but other than that, it’s unremarkable.

The thing to notice is the two functions getUserName() and getUserHair(). You’ll see with getUserName() the function parameter being passed to it is actually the destructured version of the property name from the object it receives.

So when the whole girl object is passed to the function, it returns just the property and value of name as the function’s output.

The second function, getUserHair() is even more interesting because the value it’s trying to access is actually located two levels down within the object being passed to the function, so first, the property of hair must be accessed, and then the properties unique to hair, which are type and color can be accessed.

That function will then return a string stating the object’s hair color and hair type as the output when it’s called with the object of girl.

This is also an example of how to access nested objects using destructuring, which I’ll talk about next.

Nested object destructuring

This was a topic that took me some time to wrap my head around (and to be honest, I usually still have to go back and look at the documentation again when I want to destructure multi-level nested objects).

The basic gist though is: if your object is more than one level deep within your object, you must first access its parent property, and its parents’ parent property, and so on, until you reach the outermost object property value.

Example of deeply nested object destructuring

const girl = {
  name: "Paige",
  age: 30,
  eyeColor: "blue",
  hobbies: {
    primaries: [
      {
        mostFavorite: [
          "drawing",
          "art"
        ],
        frequentlyDone: "cooking",
        relaxing: {
          reading: "fictionBooks"
        }
      }
    ]
  }
}

const {
  hobbies: {
      primaries: [
      {
        mostFavorite
      }
    ]
  }
} = girl;

console.log(mostFavorite[0]); // "drawing"
console.log(mostFavorite[1]); // "art"

const {
  hobbies: {
    primaries: [
      {
        relaxing: {
          reading
        }
      }
    ]
  }
} = girl;

console.log(reading); // "fictionBooks"

I used the same girl object from the previous example but I added the property of hobbies to the object and added some new arrays and objects within them so I could show how you can pull values out of either.

The first new object I create pulls out the nested object property of mostFavorite, which happens to be an array with two values. To reach these values, first, I have to wrap the outermost property of girl, which is hobbies in curly braces. Next, I have to wrap hobbies" property of primaries. Then, I must dive into both the array and the object that primaries contains to reach the mostFavorite property which actually holds the values I seek.

From there, it’s a simple exercise to log out any values that mostFavorite has.

In the same vein, to reach the value of the property reading buried deep within the girl object, I have to again start out by wrapping hobbies in curly braces, then proceed on to primaries, dive into the array and find the object property of relaxing and finally wrap the property of reading, which belongs to the parent object of relaxing.

Then, I can simply call reading as its own variable and get back the value nested deeply in the girl object.

It definitely takes some practice to get the hang of, but look at how much less syntax is needed to get those values than before.

So long const reading = girl.hobbies.primaries[0].relaxing.reading;. I won’t miss that.

If you’d like to read more about nested object destructuring, I wrote another article specifically about it a few months back, as well as ways to avoid undefined errors if values weren’t available. Here’s the link to it. 😄

Rest syntax in object destructuring

Last example, which is still in ECMAScript stage 4 proposal, I might add, at the time of writing this article: rest syntax plus object destructuring.

I showed this in the very first array destructuring demo, but I haven’t shown it with object destructuring yet. Turns out, it works about the same for objects as it does for arrays.

Example of object destructuring with the rest syntax

let myObjectOfNums = {
  ex: "ten",
  why: "twenty",
  zed: "thirty",
  dee: "forty",
  ee: "fifty"
}

let {ex, why, zed, ...allOthers} = myObjectOfNums;

console.log(ex); // "ten"
console.log(why); // "twenty"
console.log(zed); // "thirty"
console.log(allOthers); // { dee: "forty", ee: "fifty" }

See how easy it is to pull out the first three properties and their values from the object myObjectOfNums, as well as using the rest parameter to keep the other properties contained together in a new object called allOthers?

And the other rules of destructuring still apply just the same here. If you wanted to change the variable names from ex or why to a and b you could do so just the same as before.

Example of object destructuring with the rest syntax AND variable reassignment

let { ex: a, why: b, zed, ...allOthers } = myObjectOfNums;

console.log(a); // "ten"
console.log(b); // "twenty"
console.log(zed); // "thirty"
console.log(allOthers); // { dee: "forty", ee: "fifty" }

This is also totally valid. Pretty cool, huh? 😃

Conclusion

Array and object destructuring is something that's existed in languages like Perl and Python for years, but it wasn’t until ECMAScript 2015 that JavaScript began to get some parity in this area.

This new syntax utilizing curly braces makes it possible to easily access individual variables from arrays, and even objects, with very concise code. And I, for one, am a big fan of it.

My goal with this series is to examine parts of the ES6 syntax you use everyday but may not know all the subtleties and nuances of, so you can be an even better web developer.

Check back in a few weeks, I’ll be writing about more JavaScript and ES6 or something else related to web development.

Thanks for reading, I hope I’ve made array and object destructuring a little more clear and something you’re excited to try out in your own code bases.

References & Further Resources

Destructuring assignments, MDN docs

Object Keys, Values, and Entries: JavaScript ES6 Feature Series (Pt 9)

Tue, 12 Nov 2019 00:00:00 GMT

</figure>

Introduction

The inspiration for these pieces is simple: it’s the fact that there are still plenty of developers for whom JavaScript is sometimes completely confusing— or, at least, doesn’t always make a whole lot of sense.

But JavaScript still powers almost 95% of the 10 million most popular webpages, according to Wikipedia.

And since it only continues to increase in usage as time goes by, I wanted to provide a bunch of articles and examples of ES6+ features that I use regularly, for other developers to reference.

This piece will focus on JavaScript objects and some handy new methods concerning keys, values, and the combination of the two that were standardized into the language in the last few years.

Object.keys()

This may sound hard to believe, especially if you’re coming from another programming language like Java, but up until ES2015, there was no built-in method in JavaScript to easily access an object’s property names, otherwise known as its keys.

Really?

Shocking, I know. Prior to this, a library like Lodash might be employed to do something that should be a basic, standardized method.

Happily, the ECMAScript committee got their act together with the release of ES2015, and the method Object.keys() was introduced. The method returns an array of a given object's own enumerable property names, in the same order as we get with a normal for loop.

Example of Object.keys() on an object:

const resistanceFighter = {
  name: "John Connor",
  age: 30,
  title: "Resistance Leader",
  fight() {
    return `${this.name} leads the resistance fight against the machines.`
  }
};

console.log(Object.keys(resistanceFighter)); // [ "name", "age", "title", "fight()" ]

The Object.keys() syntax is very simple: You pass an object, like resistanceFighter, into the method as its parameter, and the method returns the keys, or names, of that object’s properties and methods.

In this case, it returns the three property names: name, age, and title, plus the method attached to the object, fight().

The elements are returned as strings in an array that correspond to the enumerable properties found directly on the object. The ordering of the properties is the same as that given by manually looping over the properties of the object.

Let’s take a look at some other examples of how Object.keys() will interact with various JavaScript data structures.

Object.keys() with a simple array:

const resistanceTeam = [ "Sarah J Connor", "Terminator", "John Connor"];

console.log(Object.keys(resistanceTeam)); // [ "0", "1", "2" ]

When running the method against an actual array, the keys returned are the “stringified” indexes of each item contained in the array.

Object.keys() with an array-like object:

const movieTitles = {
  0: "The Terminator",
  1: "Judgment Day",
  2: "Rise of the Machines",
  3: "Salvation",
  4: "Genesis",
  5: "Dark Fate"
}

console.log(Object.keys(movieTitles)); // [ "0", "1", "2", "3", "4", "5" ]

Similar to a true array, an object that’s like an array with index numbers listed as its keys will return an array of numbers as strings when it’s run through the Object.keys() method.

Object.keys() on an array-like object with random key ordering:

const movieReleaseYears = {
  1991: "Judgment Day",
  2015: "Genisys",
  2003: "Rise of the Machines",
  1984: "The Terminator",
  2019: "Dark Fate",
  2009: "Salvation"
}

console.log(Object.keys(movieReleaseYears)); // [ "1984", "1991", "2003", "2009", "2015", "2019" ]

If you run Object.keys() on an object with randomly ordered keys, like, say, keys that happen to be the years the different Terminator movies were released, the array of keys produced will be ordered from least to greatest.

If, however, you run this same method on an object where the keys are non-numerical strings, they will not be printed out in a sorted order. To get them sorted, you’ll still need to run the .sort() method on the array. See the example below for reference.

Object.keys() on an object with keys, both unsorted and sorted:

const movieDirectorsUnordered = {
  director2: "Tim Miller",
  director4: "Jonathan Mostow",
  director1: "James Cameron",
  director3: "McG"
}

console.log(Object.keys(movieDirectorsUnordered)) // [ "director2", "director4", "director1", "director3" ]
console.log(Object.keys(movieDirectorsUnordered).sort()); // [ "director1", "director2", "director3", "director4" ]

One more thing you can do with Object.keys() in ES2015: You can coerce non-object things like strings to objects.

Object.keys() coercing a string to an object:

const animal = "canine";

console.log(Object.keys(animal)); // [ "0", "1", "2", "3", "4", "5" ]

If you try to call this method in ES5 or below, a TypeError will be thrown, just as an FYI.

Now, let’s move on to the other half of the object equation: the values that go with the keys.

Object.values()

For whatever reasons, the method Object.values() wasn’t standardized in the JavaScript language until ES2017 (Why not sooner?)

The Object.values() method returns an array of a given object's own enumerable property values, in the same order as that provided by a for...in loop (the difference being that a for-in loop enumerates properties in the prototype chain as well).

If you’ve ever run a relatively strict set of ESLint rules, like Airbnb’s, against your repo, you may have seen a warning about using for...in loops for the reason mentioned above.

Looping through the prototype chain will reveal inherited properties as well as properties that belong to the object itself, which is usually not desired.

Example of Object.values() on an object:

const resistanceFighter = {
  name: "John Connor",
  age: 30,
  title: "Resistance Leader",
  fight() {
    return `${this.name} leads the resistance fight against the machines.`
  }
};

console.log(Object.values(resistanceFighter)); // [ "John Connor", 30, "Resistance Leader", [Function: fight] ]

Using the same resistanceFighter object I used in the Object.keys() example above, when I call Object.values() and pass it as the parameter, I get back an array of the values in the object: "John Connor", 30, "Resistance Leader", and the fight() function.

Unlike the array of strings that Object.keys() returns, Object.values() returns an array whose elements are the actual enumerable property values found on the object. Once again, the ordering of the properties is the same as that given by looping over the property values of the object manually.

Here are some examples of Object.values() in action.

Object.values() with a simple array:

const resistanceTeam = [ "Sarah J Connor", "Terminator", "John Connor"];

console.log(Object.values(resistanceTeam)); // [ "Sarah J Connor", "Terminator", "John Connor" ]

Just as when Object.keys() prints out the index of each item in the array, calling Object.values() prints out the actual items in the array, just as they are currently listed.

Object.values() on an array-like object:

const movieTitles = {
  0: "The Terminator",
  1: "Judgment Day",
  2: "Rise of the Machines",
  3: "Salvation",
  4: "Genisys",
  5: "Dark Fate"
}

console.log(Object.values(movieTitles));
/* [ "The Terminator",
  "Judgment Day",
  "Rise of the Machines",
  "Salvation",
  "Genisys",
  "Dark Fate" ] */

When Object.values() is called on the array-like object with numbers as its keys, the resulting array is printed out in the order of the keys. This can be proven by the following example.

Object.values() on an array-like object with random key ordering:

const movieReleaseYears = {
  1991: "Judgment Day",
  2015: "Genisys",
  2003: "Rise of the Machines",
  1984: "The Terminator",
  2019: "Dark Fate",
  2009: "Salvation"
}

console.log(Object.values(movieReleaseYears));
/* [ "The Terminator",
  "Judgment Day",
  "Rise of the Machines",
  "Salvation",
  "Genisys",
  "Dark Fate" ] */

As you can see, even though the movieReleaseYears's keys are in a random order, when the resulting array is printed out from Object.values(), the values are in the numerical order of their keys’ values.

This same sorting does not hold true when the object’s keys are non-numerical values.

In that case, as with the Object.keys() example, you must manually sort the resulting array yourself with .sort(). The array will sort according to the values in it, not the keys the values were once attached to.

Object.values() on an object’s values, both unsorted and sorted:

const movieDirectorsUnordered = {
  director2: "Tim Miller",
  director4: "Jonathan Mostow",
  director1: "James Cameron",
  director3: "McG"
}

console.log(Object.values(movieDirectorsUnordered)); // [ "Tim Miller", "Jonathan Mostow", "James Cameron", "McG" ]
console.log(Object.values(movieDirectorsUnordered).sort()); // [ "James Cameron", "Jonathan Mostow", "McG", "Tim Miller" ]

And just like Object.keys(), Object.values() has the power to coerce non-object primitives like strings into objects.

Behold, Object.values() coercing a string to an object:

const animal = "canine";

console.log(Object.values(animal)); // [ "c", "a", "n", "i", "n", "e" ]

OK, time to go to part three of this piece: the combination of object keys and values, courtesy of Object.entries().

Object.entries()

The Object.entries() method was also introduced with ES2017, and the method returns an array of a given object’s own enumerable string-keyed property [key, value] pairs, in the same order as that provided by a for...in loop.

As I stated above in the Object.values() section, a for-in loop, which could achieve a similar outcome as Object.entries(), iterates over the prototype chain as well, which isn’t usually the desired behavior of the code. This method provides a more recommended way to iterate through an object’s keys and values.

The order of the array returned by Object.entries() does not depend on how an object is defined. If there is a need for certain ordering then the array should be sorted first like Object.entries(obj).sort((a, b) => b[0].localeCompare(a[0]));.

Example of Object.entries() on an object:

const resistanceFighter = {
  name: "John Connor",
  age: 30,
  title: "Resistance Leader",
  fight() {
    return `${this.name} leads the resistance fight against the machines.`
  }
};

console.log(Object.entries(resistanceFighter));
/* [ 
    [ "name", "John Connor" ],
    [ "age", 30 ],
    [ "title", "Resistance Leader" ],
    [ "fight", [Function: fight] ] 
   ] */

for(let [key, value] of Object.entries(resistanceFighter)){
  console.log(`${key}: ${value}`);
}
/* name: John Connor
  age: 30
  title: Resistance Leader
  fight: function fight() {
    return this.name + " leads the resistance fight against the machines.";
} */

Object.entries(resistanceFighter).forEach(([key, value]) => console.log(`${key}: ${value}`))
/* name: John Connor
  age: 30
  title: Resistance Leader
  fight: function fight() {
    return this.name + " leads the resistance fight against the machines.";
} */

In the example above, calling Object.entries() on the resistanceFighter object produces an array of nested arrays, each inner array corresponding to one of the object’s key-value pairs.

The second and third parts of the example are a way to elegantly iterate through an object’s values with a for...of loop or the Array.forEach() method, both of which I find particularly useful when I need to debug or transform an object for future use.

Let’s look at some other uses for Object.entries().

Object.entries() with a simple array:

const resistanceTeam = [ "Sarah J Connor", "Terminator", "John Connor"];

console.log(Object.entries(resistanceTeam));
/* [ 
    [ "0", "Sarah J Connor" ],
    [ "1", "Terminator" ],
    [ "2", "John Connor" ] 
   ] */

Calling the Object.entries() method on a simple array will print out all the values in the array with their corresponding index values as their keys — pretty straightforward and similar to calling Object.keys() on an array.

Object.entries() on an array-like object:

const movieTitles = {
  0: "The Terminator",
  1: "Judgment Day",
  2: "Rise of the Machines",
  3: "Salvation",
  4: "Genisys",
  5: "Dark Fate"
}

console.log(Object.entries(movieTitles));
/* [ 
    [ "0", "The Terminator" ],
    [ "1", "Judgment Day" ],
    [ "2", "Rise of the Machines" ],
    [ "3", "Salvation" ],
    [ "4", "Genisys" ],
    [ "5", "Dark Fate" ] 
   ] */

Similarly, calling the method on an object with index numbers as its names, it will produce an array of arrays with the keys and values of each object property.

Object.entries() on an array-like object with random key ordering:

const movieReleaseYears = {
  1991: "Judgement Day",
  2015: "Genisys",
  2003: "Rise of the Machines",
  1984: "The Terminator",
  2019: "Dark Fate",
  2009: "Salvation"
}

console.log(Object.entries(movieReleaseYears));
/* [ 
    [ "1984", "The Terminator" ],
    [ "1991", "Judgment Day" ],
    [ "2003", "Rise of the Machines" ],
    [ "2009", "Salvation" ],
    [ "2015", "Genisys" ],
    [ "2019", "Dark Fate" ] 
   ] */

As in previous pieces of this blog, Object.entries() on movieReleaseYears, with its currently unordered number keys, will return a list of arrays sorted by their index values in ascending order.

Object.entries() on an object’s values, both unsorted and sorted:

const movieDirectorsUnordered = {
  director2: "Tim Miller",
  director4: "Jonathan Mostow",
  director1: "James Cameron",
  director3: "McG"
}

console.log(Object.entries(movieDirectorsUnordered));
/* [ 
    [ "director2", "Tim Miller" ],
    [ "director4", "Jonathan Mostow" ],
    [ "director1", "James Cameron" ],
    [ "director3", "McG" ] 
   ] */

console.log(Object.entries(movieDirectorsUnordered).sort());
/* [ 
    [ "director1", "James Cameron" ],
    [ "director2", "Tim Miller" ],
    [ "director3", "McG" ],
    [ "director4", "Jonathan Mostow" ] 
   ] */

But, if the object being made into an array of arrays has keys that are strings or something besides numbers that are easy to order, it will just print out the contents as they are listed in the object unless you manually call the .sort() method afterward. Fair enough?

Object.entries() coercing a string to an object:

const animal = "canine";
console.log(Object.entries(animal));
/* [ 
    [ "0", "c" ],
    [ "1", "a" ],
    [ "2", "n" ],
    [ "3", "i" ],
    [ "4", "n" ],
    [ "5", "e" ] 
   ] */

A final example. A string can be mutated into an object with the help of Object.entries().

As you see in the code above, the string "canine" becomes an array of arrays, with each character in the string becoming the value portion of the array and the index of that string becoming the key portion.

Conclusion

Surprisingly, until the releases of ES2015 and ES2017 JavaScript objects lacked some seemingly basic methods to read and manipulate keys and values. Luckily for us, accessing an object’s keys, its values, and even iterating through those keys and values together is now standard.

Keeping up with all the improvements the ECMAScript committee is coming out with can be more than a little overwhelming, which is why I started writing these pieces.

My goal with this series is to go in-depth on pieces of the ES6 syntax you may use daily (or may not even be aware of), so you can be an even more effective web developer.

Check back in a few weeks, I’ll be writing about more JavaScript and ES6, or something else related to web development.

Thanks for reading, I hope you’ll give ES6's newest object methods a spin in your JavaScript code — it will make your development life easier, I promise.

References & Further Resources

Object.keys(), MDN docs
Object.values(), MDN docs
Object.entries(), MDN docs

Classes and Inheritance: JavaScript ES6 Feature Series (Pt 8)

Sun, 27 Oct 2019 00:00:00 GMT

</figure>

Introduction

The inspiration behind these pieces is simple: there are still plenty of developers for whom JavaScript is a bit baffling— or at the very least, has some odd quirks in its behavior.

Despite that, JavaScript is responsible for powering almost 95% of the 10 million most popular web pages, according to Wikipedia.

Since it only continues to increase in usage and popularity, I wanted to provide a bunch of articles and examples of ES6+ features that I use regularly, for other developers to reference.

The aim is for these pieces to be short, in-depth explanations of various improvements to the language that I hope will inspire you to write some really cool stuff using JS. Who knows? You might even learn something new along the way. 😄

For my eighth blog post, I’ll dive into JavaScript classes and class-based inheritance: the cleaner, more straightforward way to handle JavaScript’s existing prototype-based inheritance system.

Before there were classes, there were prototypes

I can’t talk about JavaScript classes without first talking about prototypes — the original mechanism by which JavaScript objects inherit features from each other.

It’s outside the scope of this blog do to a deep dive into the prototype object and how prototype-based inheritance works, but I’ll give you a brief overview.

Each object in JavaScript has a prototype object, which acts as a template that it inherits methods and properties from. That same prototype object may also have a prototype object that it inherits from as well, and so on, and so forth. This is referred to sometimes as the prototype chain.

When a property or method is called on an object, the browser first checks to see if the actual object has the method on it, if not it checks if the method is available to it through its prototype object. If the method’s not defined on the object’s personal constructor (where its prototype object comes from), the browser checks up another level to see if the constructor’s prototype object has the method available to it. The browser will do this all the way up until it reaches the object at the top with null as its prototype.

</figure>

Hmm…?

Check out this example, this should help illustrate what I’m saying.

Example of traditional prototype-based inheritance

function Superhero(superName, realName, powers){
  this.superName = superName,
  this.realName = realName,
  this.powers = powers
}

const wonderWoman = new Superhero("Wonder Woman", "Diana Prince", "Strength and flight");
console.log(wonderWoman); 
/* Superhero {
  superName: "Wonder Woman",
  realName: "Diana Prince",
  powers: "Strength and flight" } */

Superhero.prototype.equipment = "Lasso of truth";
console.log(wonderWoman.realName); // Diana Prince
console.log(wonderWoman.equipment); // Lasso of truth
console.log(wonderWoman.catchPhrase); // undefined
console.log(wonderWoman.hasOwnProperty("equipment")); // false
console.log(Superhero.hasOwnProperty("equipment")); // false
console.log(Superhero.prototype.hasOwnProperty("equipment")); // true

Above in the example is a constructor called Superhero, which defines a new super-powered crime fighter.

I created wonderWoman, and gave her the properties of superName, realName, and powers. Then, I added a property called equipment to the Superhero's prototype object, with a value of "Lasso of truth".

When I called the object’s realName property, "Diana Prince” is found on the object itself. When I called equipment, "Lasso of truth” is found on the object constructor’s prototype. And when I try to call the property catchPhrase, it comes back as undefined because nowhere in the prototype chain does that property exist.

It’s proved by the lines where I check if either wonderWoman.hasOwnProperty() and Superhero.hasOwnProperty() of "equipment", and both return false, but Superhero.prototype.hasOwnProperty("equipment") returns true.

That is JavaScript’s prototype-based inheritance in a nutshell. Keep in mind, this is what’s happening in the background when we’re using class-based syntax going forward.

Now let’s get on to the new ES6 classes.

Then came classes

Unlike other object oriented programming (OOP) languages like Java, which have always been class-based, JavaScript chose prototypes for its way of handling inheritance, but with the release of ECMAScript 2015, classes were introduced to the language’s syntax.

Let me be absolutely clear though: classes are just syntactical sugar over JavaScript’s existing prototype-based inheritance. They are NOT a new object-oriented inheritance model.

Classes are what might be defined as "special functions", and just like with all functions, which I wrote about here, they can defined in two ways: class expressions and class declarations.

Class declarations

The first way to define a class is using a class declaration. To declare a class, you use the class keyword with the name of the class.

Example of class declaration

class Book {
  constructor(title, author) {
    this.title = title;
    this.author = author;
  }
}

const novel = new Book("Moby Dick", "Herman Melville");
console.log(novel); // Book { title: "Moby Dick", author: "Herman Melville" }

One important thing to note is that unlike function declarations, class declarations ARE NOT hoisted. You must first declare your class, and then access it, otherwise you’ll get a ReferenceError thrown.

Class expressions

The other way to define a class is with a class expression. Class expressions can be named or unnamed. If a class expression is named, the name given to a named class expression is local to the class’s body.

Examples of class expressions, both unnamed and named

// unnamed class expression
let Drama = class {
  constructor(title, author){
    this.title = title;
    this.author = author;
  }
};
console.log(Drama.name); // Drama

// named class expression
let Comedy = class Book2 {
  constructor(title, author){
    this.title = title;
    this.author = author;
  }
};
console.log(Comedy.name); // Book2

Ok, the basics of creating classes are out of the way, time to dive into the nitty-gritty details.

Class body & method definitions

Similar to functions once again, the body of the class is the part contained within the {}. In here is where the class members like the constructor and methods are defined.

Strict mode

The first thing you need to know is that the body of a class is always executed in strict mode.

All this means is code written here is subject to stricter syntax for increased performance, some otherwise silent errors will be thrown, and certain keywords are reserved for future versions of ECMAScript.

If you’ve been using any of the newer React, Angular or Vue frameworks, you’ve probably already been developing in strict mode by default, so it shouldn’t pose too much of a shift in your way of developing.

The constructor

Next up in the anatomy of a class is, potentially, a constructor. The constructor method is a special method for creating and initializing an object created with a class. There can only be one special method with the name "constructor" in each class. If a class contains more than one occurrence of a constructor method, a SyntaxError will be thrown.

A constructor can also use the super keyword to call the constructor of the super class, but I’ll get to that in more detail later in this post.

Prototype methods

Prototype methods can also be known as method definitions, and they’re a shorthand for a function assigned to the method’s name. These tend to make up the bulk of the class’s body.

Example of a prototype method on a class

class Book {
  constructor(title, author){
    this.title = title;
    this.author = author;
  }

  publicizeBook() {
    return `This book ${this.title} is written by renowned author ${this.author}.`;
  }
}

const novel = new Book("Harry Potter", "J.K. Rowling");
console.log(novel.publicizeBook()); // This book Harry Potter is written by renowned author J.K. Rowling.

console.log(Book.prototype.hasOwnProperty("publicizeBook")); // true

For this class Book, the method publicizeBook() is defined on it, so any objects created with the class of Book, will automatically also have the prototype method publicizeBook() available to them.

And check out the last line in the example. I check if the Book class owns the publicizeBook() method by testing if the Book's prototype object .hasOwnProperty("publicizeBook"), and it returns true. Which shows it’s actually prototype-based inheritance happening underneath at all.

Starting to make sense now?

Static methods

When you use the static keyword, it can be applied to either a method or a class. Static methods are called without instantiating their class and cannot be called through a class instance. These types of methods are often used to create utility functions for an application.

Example of a static method on a class

class Book {
  constructor(title, author){
    this.title = title;
    this.author = author;
  }

  static youMightLike(title, similarTitle) {
    return `If you like ${title}, you might also like ${similarTitle}.`
  }
}

const novel = new Book("Moby Dick", "Herman Melville");
console.log(novel.youMightLike); // undefined

console.log(Book.youMightLike(novel.title, "A Midsummer Night's Dream")); // If you like Moby Dick, you might also like A Midsummer Night's Dream.

With this example, if you try to call the static method youMightLike() on the actual class-based object novel, you only get back an undefined value. If you call it, however, on the class Book, with two parameters, you’ll get back a response recommending one book based on another book.

Beware of boxing with prototype & static methods

When a method, either static or not, is called without a value for this assigned to it, the this value will be undefined inside the method. It’s because the body of the class always runs in strict mode regardless of whether it’s explicitly set or not.

Example of undefined ‘this’ inside a class

class Dog {
  eat() {
    return this;
  }
  
  static speak() {
    return this;
  }
}

let shibaInu = new Dog();
console.log(shibaInu.eat()); // Dog {}
let chowDown = shibaInu.eat;
console.log(chowDown); // undefined
console.log(Dog.speak.toString()); // Class Dog
let greet = Dog.speak;
console.log(greet); // undefined

If you write the above class in the traditional function-based syntax, then autoboxing in method calls will happen in non–strict mode based on the initial this value. If the initial value is undefined, this will be set to the global object.

Example of this taking the global scope if it’s not assigned the function

function Dog() { };

Dog.prototype.eat = function() {
  return this;
}

Dog.speak = function() {
  return this;
}

let husky = new Dog();
console.log(husky.eat()); // Dog {}

let munch = husky.eat;
console.log(munch()); // global object (long list of available options)

let bark = Dog.speak;
console.log(bark()); // global object again (long list of available options)

Just be aware of this little difference between classes and functions, so it doesn’t unwittingly trip you up for a few hours.

Instance properties

Instance properties must be defined inside of class methods.

Example of defined instance properties inside the class

class Cat {
  constructor(eats, sleeps){
    this.eats = eats;
    this.sleeps = sleeps;
  }
  
  knocksThingsOver(obj) {
    return `Woops, the cat just knocked ${obj} over...`
  }
}

While static data properties and prototype data properties must be defined outside of the ClassBody declaration:

Example defining static data & prototype data properties

Cat.plays = true;

Cat.prototype.eatsCatnip = "Goes nuts for it";

Field declarations (still experimental)

These next features: public and private field declarations are still in the experimental feature (stage 3), proposed at TC39, by the JavaScript standards committee. But they’re still worth knowing about as they’ll probably make it into the standard syntax sometime soon.

Native support in browsers is limited as of now, but the feature can be used through a build step with systems like Babel, courtesy of your Webpack config.

Public declarations

With the JavaScript field declaration syntax, the previous example of Cat can be written like so.

Example of public declarations in a class

class Cat {
  eats = "mice";
  sleeps;

  constructor(eats, sleeps){
    this.eats = eats;
    this.sleeps = sleeps;
  }

  knocksThingsOver(obj) {
    return `Woops, the cat just knocked ${obj} over...`
  }
}

By declaring fields up-front, class definitions become more self-documenting, and the fields are always present. It’s also cool that the fields can be declared with or without a default value, just like default values in functions, which I wrote about here.

Private declarations

Private declarations aren’t too different syntactically from their public counterparts. But they are slightly different in usage and execution.

Private fields can only be declared up-front in a field declaration. They cannot be created later through assigning to them, the way normal properties can.

It’s also an error to reference private fields from outside of the class; they can only be read or written within the class body. By defining things which are not visible outside of the class, you ensure that your classes’ users can’t depend on internals, which may change version to version.

Example of private declarations in a class

class Cat {
  #eats = "mice";
  #sleeps;

constructor(eats, sleeps){
    this.#eats = eats;
    this.#sleeps = sleeps;
  }

knocksThingsOver(obj) {
    return `Woops, the cat just knocked ${obj} over...`
  }
}

Like I said before, not widely in use yet, and I probably wouldn’t put it into production, but keep an eye out for these in future ES releases.

Sub classing with `extends`

The extends keyword is used in class declarations or class expressions to create a class as a child of another class.

Example of extending one class into a child as a sub class

class Animal {
  constructor(name) {
    this.name = name;
  }
  speak() {
    console.log(`${this.name} makes a noise`);
  }
}

class Horse extends Animal {
  constructor(name) {
    super(name);
  }
  speak() {
    console.log(`${this.name} whinnies.`)
  }
}

let thoroughbred = new Horse("Seabiscuit");
console.log(thoroughbred.speak()); // Seabiscuit whinnies.

The class Horse extends the other class Animal, to take in its methods, and since Horse also has its own constructor present, it needs to call super() before referencing this.

This syntax of extends is very similar to React’s class-based components, if you’ve ever worked with that framework.

React class-based component syntax

import React, { Component } from "react";

class Nav extends Component {
  // ...do something
}

It’s also worth noting, when methods for a parent and child class are named similarly, the child’s method takes precedence over the parent’s method when calling it.

Super class calls with `super`

Now we’re back to that super keyword that’s been mentioned above a few times. The super keyword is used to call corresponding methods of super class. This is one distinct advantage over prototype-based inheritance, because technically, both methods from the parent and child class could be invoked just by using super instead of one or the other.

Example of accessing the parent class methods from the child class

class Animal {
  constructor(name) {
    this.name = name;
  }

  speak() {
    console.log(`${this.name} makes a noise`);
  }
}

class Hippo extends Animal {
  speak() {
    super.speak();
    console.log(`${this.name} roars.`);
  }
}

let hippo = new Hippo("Bertha");
console.log(hippo.speak()); // Bertha makes a noise.
// Bertha roars.

In this example, Hippo's speak() method actually calls both Animal's speak() method as well as its own unique one, by simply using super.speak() inside of Hippo's speak().

And that’s really all there is to super, it’s not so complicated once some examples are shown. Right?

And this, is really the most important parts about classes you’ll need to know while developing in JavaScript. Well done for making it to the end of this article! 😃

Conclusion

Classes have existed in other programming languages for decades, and JavaScript finally got on board (sort of) with the release ES 2015. Although classes in JavaScript look the same syntactically as other OOP languages, it’s still really prototype-based inheritance taking place under the hood.

This new way of thinking about methods and properties may take a little getting used to for all of us familiar with traditional prototype object, but classes provide some really great improvements to make our development lives easier, and I hope you’ll try them out to see for yourself.

My aim with this series is to deeply explain pieces of the ES6 syntax you use every day so you can use them for maximum impact and avoid common pitfalls.

Check back in a few weeks, I’ll be writing about more JavaScript and ES6 or something else related to web development.

Thanks for reading, I hope you’ll give ES6 class-syntax a chance in your JavaScript for easier prototype inheritance.

References & Further Resources

Object prototypes, MDN Docs
Classes, MDN docs

Built-in Module Imports and Exports: JavaScript ES6 Feature Series (Pt 7)

Mon, 14 Oct 2019 00:00:00 GMT

</figure>

Introduction

The inspiration behind these pieces is simple: There are still plenty of developers for whom JavaScript doesn’t make a whole lot of sense — or at the very least, is perplexing at times.

According to Wikipedia, JavaScript powers just under 95% of the 10 million most popular web pages, as of May 2017.

Since JS powers so much of the web, I wanted to provide a slew of pieces and examples of ES6+ features that I use regularly, for other developers to reference.

The aim is for these pieces to be short, in-depth explanations of various improvements to the language that I hope will inspire you to write some really cool stuff using JS. Who knows? You might even learn something new along the way.

Piece seven in my series concerns built-in module support in JavaScript, the easier way to make objects, functions, classes, and variables available anywhere in your codebase, with minimal syntax.

Require is replaced by ES modules

Before I dive into built-in modules, let me give you a very brief history lesson on CommonJS and the require syntax.

Pre-ES2015, the most widely used approach to adding various library modules to JavaScript files’ scope was via CommonJS. Node.js uses it, and it’s the system used by most packages on npm today. The main concept in CommonJS modules is a function called require(). When you call this with the module name of a dependency, it makes sure the module is loaded and returns its interface to be used in the file’s scope.

If you’ve ever worked with Node.js scripts, or even older client-side JavaScript files, you’ve likely seen this syntax before. Here’s an example that imports ExpressJS, a widely used server-side framework, into a Node.js app.js file:

Anatomy of CommonJS's require

const express = require("express");

Simply by declaring the variable express and calling require("express"), all of ExpressJS’s various methods are now available to the file.

For a long while, CommonJS modules worked quite well and allowed the JavaScript community, via npm, to share code on a large scale.

But CommonJS has a few quirks that make it a less than ideal solution: namely, the fact that require() is a normal function call taking any kind of argument, not just a string literal, which makes it hard to determine the dependencies of a module without running its code. There are more issues than this, but I won’t get into them here. This is a brief history, after all.

Suffice it to say, ES2015 introduced its own module system, and in doing so, integrated the new module support notation into the language. The main concepts of dependencies and interfaces remain the same, but the details differ — which is what I’ll be discussing now.

NOTE: If you’d like a deeper dive into modules, packages, and CommonJS, I’d recommend reading this chapter out of Eloquent JavaScript.

But for now, let’s get on with ES6’s built-in module support.

Import all the things

One of the biggest departures from CommonJS’s module syntax, when ES modules came to town, is the special new keyword import to access a dependency.

Let’s go through all the different types of imports that are available. In the section below this, I’ll go through the export options as well.

Import default exports

The very first type of import revolves around default imports. This is the most frequently used type of import, and there are a few different ways default exports can be combined with other types of imports.

Simplest example importing only the default export:

import carTires from "./autoParts/tires.js";

In this first example, carTires is the default export from the file tires.js. To get access to it in this file, all that needs to be done is to declare the export by the same name.

If a file has more than one value you’d like access to (like say, a function or variable, plus a default export) the syntax would look like the following example of import of a default export and an additional function:

import carTires, { shineTires } from "./autoParts/tires.js";

The destructured export { shineTires } is what’s known as a named import in this particular import statement, since it’s wrapped in curly braces and exported by the same name from the tires.js file.

There’s one other option where default exports can be combined with an entire module’s contents via a wildcard import *.

Example of importing a default export plus all the other exports in a file:

import carTires, * as tireOptions from "./autoParts/tires.js";

If there was a variable called tireType and a function called rotateTires() from the file tires.js, the importing file could access them both by calling tireOptions.tireType or tireOptions.rotateTires(). Since the additional exports were imported with the syntax * as tireOptions, all exports from the file can be referenced tireOptions.xyz. This is known as a namespace import.

NOTE: If you’re importing a default export along with named or namespace imports, be aware that the default export will have to be declared first.

But I’m getting ahead of myself. Let’s move on from importing default exports to all the other import types, and then the details should become more clear.

An entire module's contents: namespace imports

As I alluded to above, if a file has lots of individual exports you’d like access to, without having to import them all by name, there’s a way to do so with the help of import * as abc or a namespace import.

This inserts abc into the current scope, containing all the exports from the module in the file located in /romanAlphabet/letters.js.

Example of importing multiple exports all at once from a file:

import * as abc from "/romanAlphabet/letters.js";

Here, accessing the exports means using the module name (abc in this case) as a namespace. For example, if the module imported above includes an export singMyAbcs(), you would call it like this: abc.singMyAbcs();

Simple enough, right? Now let’s move on.

A single export from a module: named import

The second example above, in the default export section, touched on is named imports, a simple way to bring in just one object or value into the importing file’s scope using ES6 object destructuring.

Here’s another example of a single named import to make it even more clear:

import { apples } from "./plants/fruits.js";

The object apples is exported from the file fruits.js, either implicitly because the whole module is exported or explicitly using the export statement, and is then inserted into the current scope.

This same syntax can be used for multiple imports too.

Multiple exports from a single module

Similar to a single named import, multiple values and objects from a file can be exported using the same code style.

Example of multiple named imports:

import { carrots, potatoes, onions } from "./plants/veggies.js";

All the exports from veggies.js are imported into the current file’s scope, and are accessible by name, just the same as a single imported value.

Alias an import with a more convenient name

Now suppose you have an export you’d like to import into your file, but it has a terribly long name like mySuperCaliFragilisticExpialidociusObject. That’s quite a pain to type out, right?

Well, there’s a solution: aliasing an export with a more convenient name when importing it. Check this out.

Example renaming a named export for convenience:

import { mySuperCaliFragilisticExpialidociusObject as mySuperObject } from "./maryPoppins.js"

Whenever you need to access the imported value in the current scope, you can simply call it with mySuperObject instead of the original, much longer exported name. How convenient!

Rename multiple exports during import

But say you have multiple long, named export objects. Can you rename all of them on import? Well, as it turns out, you can.

Just like renaming one export, you can rename multiple exports on import.

Example of aliasing multiple exports:

import { ladyMaryCrawley as ladyMary, ladyEdithCrawley as ladyEdith, ladyCoraCrawley as ladyGrantham } from "./downtonAbbeyFamily/ladies.js";

All of the ladies from Downton Abbey being imported into the current file have quite long titles, so, for convenience, the current scope will access them all with their shortened aliases of ladyMary, ladyEdith, and ladyGrantham.

Import side effects only

This next example I’ve never used personally, but you have the option to import an entire module for side effects only, without importing anything. This has the side effect of running the module’s global code, while not actually importing any values.

Example of importing a module without importing anything in particular:

import "./helperFunctions.js";

If anything, this syntax reminds me quite a bit of the way CSS files are now imported into individual JavaScript files.

Dynamic imports

Ok, last import to go over: dynamic imports. This proposal is due to be finalized in the ES2020 release, so I wouldn’t put it into production just yet, but it’s worth being aware of.

The import keyword may be called as a function to dynamically import a module. When used this way, it returns a promise.

Example of dynamic imports with promise syntax:

import ("./waysToTravel.js")
  .then((vehicles) => {
    // do something with planes, trains and automobiles
  });

It can also be used with the newer ES6 async/await syntax.

Example of dynamic imports with async / await syntax:

let vehicles = await import("./waysToTravel.js");

I haven’t used this type of import myself yet, but it’s quite nice that they’re working on so many different ways to access objects and values across scopes with the help of built-in module support.

Export all the other things

All right, we’ve covered the import side of ES6’s module support. Now let’s look at the other half of the equation: the ES6 export() keyword and syntax.

Default exports

If you’ve worked in ReactJS at all, you’ll probably be very familiar with default exports. If you simply place the default keyword after the variable that you’re defining for export, it becomes the default export. Therefore, you’ve excluded it from needing curly braces when being imported into another file. Let’s take a look at some examples.

Examples of a default function, variable or class export from a file:

// this is how to export a function as a default
export default function getMovies() {
  // fetch some movie data and return it
};

// this is how to export a variable as a default
export default const movie = {
 title: "The Lion King", 
 releaseDate: "July 19, 2019",
 synopsis: "Simba idolizes his father, King Mufasa, and takes to heart his own royal destiny on the plains of Africa."
};

// this is how to export an ES6 class (a React class to be exact)
export default class Movies extends Component {
  // render some movie data in JSX
}

Take note: each JavaScript file can only have one default export. All other exports in that same file will just have the export keyword in front of them.

Individual named exports

The next type of export to cover is individual named exports, which I touched on when discussing the imports of such values in the import section above.

Named exports aren’t very complicated — as I said before, anything besides a default export in a file you’d like to make available for use in other JavaScript files can be a named export.

Examples of multiple named exports in a file:

// this first array of pets is available to be imported into any other file because it has the "export" keyword
export const myPets = [ "dog", "cat", "guinea pig", "gold fish"];

// mySecretPets can't be directly accessed outside the scope of this file because it lacks the "export" keyword ahead of it
const mySecretPets = [ "dragon", "griffin", "Loch Ness monster", "Big Foot"]

// nameMyPets can be called in the scope of other files
export function nameMyPets() {
  console.log(myPets, mySecretPets)
}

As you can see from the examples above, both the variable myPets and the function nameMyPets are explicitly exported from the file and can thus be imported into another file using the named import syntax: import { myPets, nameMyPets } from "./pets.js";. The other variable mySecretPets, however, is not exported from the file and therefore cannot be accessed explicitly outside of its original scope.

Multiple named exports in a single line

There’s another way you can export multiple values from a single file in JavaScript, as well, if you prefer not to type export multiple times throughout the file.

At the end of the module, with syntax almost identical to importing multiple named imports, you can declare any values you’d like to make exportable to other scopes outside of the current file.

Example of a single line with multiple named exports:

export { cakes, cookies, makeDessert, makeTea };

If you can imagine in the example above that the variables cakes and cookies existed along with the functions makeDessert() and makeTea(), and you wanted all of those values to be accessible in other areas of your codebase, simply by exporting them all at the end of the module, you could have access to them wherever else you desired.

It’s a slightly cleaner syntax, but really, explicit exports for each value or a single export for all the values at the end of the module will achieve exactly the same thing for you.

Exporting with aliases

The final export option I’ll cover is exporting with aliases, which is similar to importing with aliases.

Example of aliasing exports:

export { dumplings, xiaoLongBao as soupDumplings, bbqPorkBuns, orderDimSum, pickUpSteamerBaskets, pourTea as fillTeaCups };

Just as with renaming named exports in the import file, you can actually rename the exports as you export them. In the example above, I renamed the variable xiaoLongBao to soupDumplings, and the function pourTea to fillTeaCups for importing into any other scope.

Honestly, I’m not really sure why you’d give a value one name in a file and then choose to export it with a different name to all other files. (If there was a naming collision in another file’s scope, why not just rename the import then?) But it’s an option, and I want to arm you with all the knowledge I can.

And this concludes our tour through the ins and outs of imports and exports via ES6’s built-in module support!

Conclusion

JavaScript’s ES6 syntax has been out for a number of years now (and gaining wider and wider adoption as time passes), but it introduced some groundbreaking changes to the language, which were too foreign for some developers to be excited about learning immediately.

And while I can agree that it’s quite different, it’s also incredibly powerful and makes writing JavaScript code so much easier than it was just a few years prior.

My aim with this series of pieces is to highlight bits of the ES6 syntax you use every day and explain how you can use these new parts of the JavaScript language for maximum impact.

Up until now, there was no standardized, built-in module support for all the wonderfully helpful JavaScript library modules out there in npm. ES2015 changed that and brought in new specialized keywords and a myriad of ways to import and export all sorts of JS values from one scope to another with a lot less fuss and bother than before. It’s been quite a game-changer.

Check back in a few weeks, I’ll be writing about more JavaScript and ES6 or something else related to web development.

Thanks for reading, I hope you’ll start to take advantage of built-in module support in your own projects if you haven’t already.

References & Further Resources

import, MDN docs
export, MDN docs
ES modules, Eloquent JavaScript

Enhanced Object Literal Value Shorthand: JavaScript ES6 Feature Series (Pt 6)

Sun, 29 Sep 2019 00:00:00 GMT

</figure>

Introduction

The inspiration behind these posts is simple: there are still plenty of developers for whom, JavaScript doesn’t make a whole lot of sense — or at the very least, is perplexing at times.

According to Wikipedia, JavaScript powers just under 95% of the 10 million most popular web pages, as of May 2017.

Since JS powers so much of the web, I wanted to provide a slew of articles and examples of ES6+ features that I use regularly, for other developers to reference.

The aim is for these articles to be short, in-depth explanations about various improvements to the language, that I hope will inspire you to write some really cool stuff using JS. Who knows, you might even learn something new along the way. 😄

My sixth post in this series will be about the new object property and method value shorthands introduced by ES2015, the most concise ways yet to initialize object properties from variables and define functions.

Objects: everything in JavaScript is one

When it comes to JavaScript, you may have heard the phrase thrown around that "everything in JavaScript is an object." Functions, strings, arrays, objects — they’re all objects. It can also be something else, but except for primitives, everything else is also an object in JS.

This article, however, concerns itself with object objects; the ones commonly initialized using the new Object(), Object.create() or literal notation initializers.

Let’s look at them before we get to the new changes courtesy of ES2015.

Object initializer examples: new Object()

// object constructor
function Pet(type, name, age, greeting) {
  this.breed = type;
  this.name = name;
  this.age = age;
  this.greeting = greeting;
  this.sayHello = function () {
    return `${this.name} says "hello" as ${this.greeting}`;
  }
}

// new Object()
const pet1 = new Pet("cat", "Felina", 3, "meow");
console.log(pet1);

/* prints: Pet {
breed: "cat",
name: "Felina",
age: 3,
greeting: "meow",
sayHello: [Function] } */

console.log(pet1.sayHello());
// prints: Felina says "hello" as meow

This first example shows how to create an object using the new Object() keywords. First, an object constructor called Pet is defined, which takes in a number of parameters, and even has its own method, sayHello() attached to it.

Then, the variable pet1 is declared and new Pet() is called with the appropriate parameters passed to it. Voilà — a new object named pet1 now exists when you call it or its method, pet1.sayHello().

Object initializer examples: Object.create()

// Object.create()
const pet2 = Object.create(pet1);
pet2.breed = "dog";
pet2.name = "Rufus";
pet2.age = 4;
pet2.greeting = "woof";
console.log(pet2);

/* prints: Pet {
breed: "dog",
name: "Rufus",
age: 4,
greeting: "woof" } */

console.log(pet2.sayHello());
// prints: Rufus says "hello" as woof

The second example above demonstrates using the Object.create() initializer. This option requires an already existing object exist as the prototype for the newly created object.

In this instance, pet2 takes pet1 from the first example as its prototype. Then, pet2 assigns all the properties from pet1 with new values, and it also inherits the method sayHello() from pet1 as well.

So when pet2.sayHello() is called, it prints out Rufus says "hello" as woof. Simple as that.

Object initializer examples: object literals {}

// object literal initialization
const pet3 = {
  type: "guinea pig",
  name: "Holly",
  age: 6,
  greeting: "snuffle",
  sayHello: function() {
    return `${this.name} says "hello" as ${this.greeting}`
  }
}

console.log(pet3);
/* prints: { type: "guinea pig",
name: "Holly",
age: 6,
greeting: "snuffle",
sayHello: [Function] } */

console.log(pet3.sayHello());
// prints: Holly says "hello" as snuffle

// object literal property and method assignment by variables
const type = "fish";
const name = "Nemo";
const age = 2;
const greeting = "bloop";
function sayHello() {
  return `${this.name} says "hello" as ${this.greeting}`;
}

const pet4 = {
  type: type,
  name: name,
  age: age,
  greeting: greeting,
  sayHello: sayHello
}

console.log(pet4);
/* prints: { type: "fish",
name: "Nemo",
age: 2,
greeting: "bloop",
sayHello: [Function: sayHello] } */

console.log(pet4.sayHello()); 
// prints: Nemo says "hello" as bloop

This final object initialization is the most frequently used one in day-to-day coding, by far: object literal initialization.

It can be done two ways:

By declaring the object and assigning all its properties and methods at the same time its initialized: like pet3.
By declaring any variables and functions ahead of time, then assigning those to a new object, like pet4.

However you choose to do it, both ways are perfectly acceptable, and frequently used.

Object literals, in particular, are what ECMAScript 2015 improves dramatically upon, let’s see how.

Shorthand property names

The first thing ES2015 improved upon regarding object literals is the abbreviated initialization of a property within an object, provided that the property key matches an existing variable name. This is known by most as: shorthand property names.

Traditional syntax for assigning variables to an object's properties

// traditional object property assignment
const breed = "chinchilla";
const howOld = 1;
const nickname = "Chi Chi";
const activities = "dust baths";

function funFact() {
  return `${this.nickname}'s fun fact is she likes ${this.activities}`;
}

const exoticPet = {
  breed: breed,
  nickname: nickname,
  howOld: howOld,
  activities: activities,
  funFact: funFact
};

console.log(exoticPet);
/* { breed: "chinchilla",
nickname: "Chi Chi",
howOld: 1,
activities: "dust baths",
funFact: [Function: funFact] } */

console.log(exoticPet.funFact()); 
// prints: Chi Chi's fun fact is she likes dust baths

Above is the traditional way to assign variables to an object’s properties. Seems a lot of extra typing in creating exoticPet when the properties already match the variable names doesn’t it?

Let’s fix that redundancy in the next example.

ES2015 shorthand example of assigning like-named variables to an object’s properties

// es2015 shorthand object property assignment
const otherBreed = "alpaca";
const firstName = "Alfie";
const likes = "frolicking";
function timeSpent() {
  return `${this.firstName} enjoys ${this.likes} whenever he can.`;
}

const exoticPet2 = {
  otherBreed,
  firstName,
  likes,
  timeSpent
};

console.log(exoticPet2);
/* { otherBreed: "alpaca",
firstName: "Alfie",
likes: "frolicking",
timeSpent: [Function: timeSpent] } */

console.log(exoticPet2.timeSpent());
// prints: Alfie enjoys frolicking whenever he can.

As you can see, the syntax for exoticPet2 is much cleaner than the previous example. Since exoticPet2’s property names match the variables declared above: otherBreed, firstName, and so on, you can simply declare the property names once and ES2015 understands how to insert the variables of the same name into the object.

Pretty neat trick, huh? I do so appreciate this particular nicety of the ES6 improvements. I use it frequently.

NOTE: Beware of using the same name for your properties, the second property will overwrite the first.

Beware: Duplicate properties assume the last value assignment

// duplicate properties assume the last value assigned

const exoticPet3 = { type: "parrot", type: "cockatoo"};

console.log(exoticPet3.type); // prints: cockatoo

Shorthand method definitions

Next up in our improvements to object literals courtesy of ES2015: method shorthand definitions. Method definition shorthand syntax omits the function keyword and colon, just like with the property assignments.

Traditional syntax to create methods (functions) in an object

// traditional method definitions
const person = {
  play: function() {
    return "I like to play board games with my friends.";
  },
  swim: function() {
    return "When I exercise, I like to swim laps in the pool.";
  }
};

console.log(person.play()); // prints: I like to play board games with my friends.

console.log(person.swim()); // prints: When I exercise, I like to swim laps in the pool.

In traditional JavaScript, methods (functions attached to objects), similar to normal object properties, are declared on an object with a property name like play or swim, and then assigned values with the traditional anonymous function declaration syntax of function() { /* ...do something here */ };.

Then, the object’s method can be called just like its properties are: person.play() or person.swim(). Nothing too earth shattering there.

But does anyone really like anonymous functions much? Nah. It doesn’t seem to add much except unnecessary code, and if we have concise arrow function expressions now for traditional functions, why not make something similar for object methods too?

ES2015 shorthand to create methods (functions) in an object

// es2015 shorthand method definitions
const person2 = {
  play() {
    return "I like to play the violin in my free time.";
  },
  swim() {
    return "Swimming in the ocean is my favorite thing to do at the beach.";
  }
};

console.log(person2.play()); // prints: I like to play the violin in my free time.

console.log(person2.swim()); // prints: Swimming in the ocean is my favorite thing to do at the beach

Introducing ES2015 method shorthands for object literal notation. The person2 object has two methods similar to person1, play(), and swim(), and as you also see, the declarations on the the object itself lack the function keyword, instead taking the property name and making that the function name instead.

Cleaner, more succinct, and to be honest, it makes more sense to me when I read the code written this way than with the anonymous functions of old.

Ok, time to move to the last ES2015 object literal improvement: computed property names.

Computed property names

Starting with ECMAScript 2015, the object initializer syntax supports computed property names. This feature allows you to put an expression in brackets [], that will be computed and used as the property name.

ES2015 shorthand for computing property keys / names via bracket notation

// computed property names / keys
let i = 1;
const items = {
  ['item' + i++] : i,
  ['otherItem' + i++] : i,
  ['aThirdItem' + i++] : i
}

console.log(items); // prints: { item1: 2, otherItem2: 3, aThirdItem3: 4 }

To be honest, I haven’t used this computed property names functionality once in daily life, but it’s nice to know it’s there if the situation should call for it, right?

And that’s it. That’s what you need to know about object literals in JavaScript as of ES2015.

Conclusion

All JS developers have gotten familiar with the language’s many quirks and gotchas over time, but lately, each year the ECMAScript committee releases new updates to the JavaScript language designed to make our lives easier. Some updated syntax changes look more foreign that others, but I’m happy to say the improvements to how we work with object literals are more similar to the older syntax than most.

Happily, the changes result in concise objects that can be more easily assigned variables, methods and even new, computed property names. It’s pretty great.

My aim with this blog series is to dive into some of my favorite parts of the JavaScript and ES6 syntax in use today, and show you how to use the best bits to maximum effect.

Check back in a few weeks, I’ll be writing about more JavaScript and ES6 or something else related to web development.

Thanks for reading, I hope you’re able to use this new object literal shorthand to make your own JavaScript objects more compact and manageable in the future.

References & Further Resources

Object literal notation, MDN docs
Object property definitions shorthand, MDN docs
Object method definitions shorthand, MDN docs

String Template Literals: JavaScript ES6 Feature Series (Pt 5)

Mon, 16 Sep 2019 00:00:00 GMT

</figure>

Introduction

The inspiration behind these pieces is simple: there are still plenty of developers for whom JavaScript is somewhat mysterious. I agree, there’s a lot to understand — it’s a complex language, once you’ve scratched the surface.

However, JavaScript powers 95% of the most popular web pages online today, according to Wikipedia, and on top of that the ECMAScript committee keeps releasing updates on a yearly basis. It’s a lot to keep up with.

Since these changes are coming so fast and furious, I wanted to provide articles and examples of ES6+ features that I use regularly, for other developers to reference.

The aim is to keep these articles short. I will provide in-depth explanations of various improvements to the language, that I hope will inspire you to write some really cool stuff using JavaScript. Who knows, you might even learn something new along the way!

In my fifth piece, I’ll be discussing template literals: the newest, easiest way to work with strings in JavaScript.

Template Literals

Strings are one of the main JavaScript objects types. All they are is sequences of characters wrapped between single-quotes '' or double-quotes "".

Anatomy of a traditional string

const string1 = 'this is a string in single quotes';

const string2 = "this is a string in double quotes";

The string type values above are immutable (unchangeable), but string objects, created by using the String() constructor, are mutable, because they are objects and you can add new properties to them.

But that’s JavaScript 101 and besides the point of this article. What I’m excited to talk about is template literals (known as "template strings" pre-ES2015).

To get us started, here’s the official MDN definition of template literals:

"Template literals are string literals allowing embedded expressions. You can use multi-line strings and string interpolation features with them." — MDN web docs

What do I mean when I say "embedded expressions" or "string interpolation features"? I’m so glad you asked, let me show you!

Template literals syntax

// just a normal string in one line - not much different from a traditional string
const simpleString = `a template literal is surrounded by backticks`;

// a string spread across multiple lines
const multiLineString = `it can be spread across
multiple lines with just 
one set of enclosing backticks`;

const name = "Paige";

// an expression placeholder allowing a variable to be injected into the string
const greetByName = `Hello, ${name}`;

console.log(greetByName); // 'Hello, Paige'

These three examples show what you can do with template literals.

The first, simpleString, is a single line string much like a normal JavaScript string surrounded by traditional quotes.

The second, multiLineString, is a multi-line string (return characters included even though you don’t see the \r characters for them), that will print out exactly the way it’s written above.

The third is an example of an expression placeholder, where the variable name is inserted into the string greetByName.

Now that I’ve shown you the quick way to use template literals, let’s get into the details of what you need to know about them.

Defining traits of template literals

Template literals have a number of characteristics that make them different from JavaScript strings of yesteryear. These same subtle differences also make them a lot more useful, in my opinion.

Here’s what makes template literals unique.

Backticks

The easiest way to identify template literals is by the backticks (``) (also known as the grave accent) surrounding the characters instead of single or double quotes.

For Mac users, the backtick key is located in the top left of the keyboard. It’s the same key you use for tildes ~ in the command line terminal.

Backtick string examples:

/* your basic JavaScript strings, but with backticks. 
notice the lack of escape characters needed for things 
like single quotes, double quotes, and apostrophes
when backticks are employed */

const fooString = `A string called 'fooString'`;

const barString = `Another string named "barString"`;

const bazString = `Without fooString and barString, you can't have bazString, right?`;

NOTE: If you need to write a backtick inside a template string, you must escape it with a backslash. \ is the same as "`".

Some JavaScript developers have even suggested using Prettier and other auto-formatters to change all their strings to use backticks all the time. I myself default to using single quotes, unless I need the power of template literals, but that’s just me!

Expression placeholders

Template literals can contain placeholders, which are indicated within the string by the dollar sign and curly braces (${variableExpression}). The expressions in the placeholders and the text between the backticks ( ) get passed to a function.

Expression placeholder examples:

function authorize(user, action) {
  if (!user.hasPermission(action)) {
    throw new Error(
      `User ${user.name} is not allowed to ${action}.`
    );
  }
}

const person = {
  name: "Sean"
};
const activity = "bake";

console.log(authorize(person, activity)); 
// "User Sean is not allowed to bake."

Multi-line strings

Any newline characters inserted in the source are part of the template literal. Instead of having to use the \n and + sign to concatenate two strings together to have a multi-line string, the whole string can be done as one.

Original way to create a multi-line string example:

console.log("To make a multi-line string this way\n" + 
"I have to concatenate two separate strings with a + sign");

/* prints: To make a multi-line string this way
I have to concatenate two separate strings with a + sign */

Template literal way to create a multi-line string example:

console.log(`With template literal backticks, I can spread strings across
as many lines as I want
with
no
problem.`);

/* prints: With template literal backticks, I can spread strings across
as many lines as I want
with
no
problem. */

Expression interpolation

Embedding expressions (like simple math calculations) into strings used to be quite cumbersome to write out (and interpret) in code.

String concatenation to insert expression values into them:

const c = 10;
const d = 5;

console.log('It used to be harder to calculate that ' + (c + d) 
+ '\n is not the same as ' + (2 * c) + ' in a string.');

/* prints: It used to be harder to calculate that 15
is not the same as 20 in a string. */

Template literals make use of syntactic sugar to make substitutions like this much more readable.

Expression interpolation inside of string template literals:

const c = 10;
const d = 5;

console.log(`With the syntactic sugar of ES6 template literals, 
doing math in strings like this: ${c + d} 
and that: ${2 * c} is a cinch.`);

/* prints: With the syntactic sugar of ES6 template literals, 
doing math in strings like this: 15
and that: 20 is a cinch. */

Nesting templates

Sometimes nesting a template is the easiest and most readable way to have configurable strings. It’s simple to allow inner backticks within a backticked template by using them inside a placeholder, ${ }, within the template. This could be useful in cases where you’re using a ternary to determine which template literal return.

Configuring strings prior to nested templates:

var classes = 'header';

function isDesktopSize(){
  window.innerWidth > 1400 ? true : false;
};

var navbar = {
  isOpen: false
};

classes += (isDesktopSize() ?
   '' : navbar.isOpen ?
     ' collapse-navbar' : ' expand-navbar');

console.log(classes);

// prints: header expand-navbar

Nested templates to create configured strings:

function isDesktopSize(){
  window.innerWidth > 1400 ? true : false;
}

var navbar = {
  isOpen: true
};

classes = `header ${ isDesktopSize() ? '' :
 `icon-${navbar.isOpen ? 'collapse' : 'expand'}` }`;

 console.log(classes);

 // prints: header icon-collapse

Nested ternaries (which are used in both configurable string examples) are a generally frowned-upon coding practice, but I used them anyway in the examples to illustrate how the classes variable can be configured so much more cleanly.

NOTE: in actual coding, it’s suggested to use if / else / if statements if you find yourself dealing with nested ternaries.

Tagged templates

More advanced forms of template literals are tagged templates.

Tags allow you to parse template literals with a function. The first argument in a tag function contains an array of string values. The remaining arguments are related to the expressions. In the end, the function can return a manipulated string (or it can return something completely different). The name of the function used for the tag can be whatever you want.

Tagged template example:

var guy = 'Paul';
var age = 96;

function myTag(strings, personExp, ageExp) {
  var str0 = strings[0]; // "That "
  var str1 = strings[1]; // " is a "

  var ageStr;
  if (ageExp > 99){
    ageStr = 'centenarian';
  } else {
    ageStr = 'youngster';
  }

  // You can even return a string built using a template literal - just to be fancy
  return `${str0}${personExp}${str1}${ageStr}`;
}

var output = myTag`That ${ guy } is a ${ age }`;

console.log(output);
// prints: That Paul is a youngster

var lady = 'Elizabeth';
var age = 107;

var output = myTag`That ${ lady } is a ${ age }`;

console.log(output);
// prints: That Elizabeth is a centenarian

Pretty neat!

Raw strings

The special raw property, available on the first argument of the tag function below, lets you access raw strings as they were entered, without processing escape sequences (\t, \v, \n, \r, etc.).

Raw property on a template literal string example:

function tag(strings) {
  console.log(strings.raw[0]);
}

tag`I'd like this line 1 \n to ignore the newline and be in line with 1 
\n even though I should be on line 3 at this point`;

/* prints: I'd like this line 1 \n to ignore the newline and be in line with 1
\n even though I should be on line 3 at this point */

// including the two characters '\' and 'n' multiple times

Additionally, the String.raw() method exists to create raw strings just like the default template function and string concatenation would.

Raw property with expression interpolation and string methods example:

var str = String.raw`What's up \n${4+7}?`;
console.log(str);
// prints: What's up \n11?

console.log(str.length);
// prints: 15

console.log(str.split('').join(','));
// prints: W,h,a,t,',s, ,u,p, ,\,n,1,1,?

I’m not sure how often you’ll need this in the real world, but it’s nice to know it’s there.

Conclusion

Every year ECMAScript releases new updates to the JavaScript language designed to make developers’ lives easier. At first glance, the syntax can seem quite alien — even to veteran JavaScript developers. However, it’s well worth your time to learn the new tricks — they’ll bring so much power to your development.

My aim with this blog series is to explain some of my favorite parts of the JavaScript and ES6 syntax being used everywhere now, and show you how to use the newest parts to maximum effect.

Strings are one of the most basic parts of the JavaScript language, but when ES2015 came around it introduced template literals and these once humble objects got a lot of upgrades. Injecting variables, performing expressions, even making multi-line strings — once complicated code is a breeze thanks to template literals.

Check back in a few weeks, I’ll be writing about more JavaScript and ES6 or something else related to web development.

Thanks for reading, I hope you’ll give template literals in their many forms a try — they make strings in JavaScript so much easier to work with.

References & Further Resources

Spread & Rest Parameters: JavaScript ES6 Feature Series (Pt 4)

Sun, 01 Sep 2019 00:00:00 GMT

</figure>

Introduction

The inspiration behind these posts is simple: there are still plenty of developers for whom, JavaScript doesn’t make a whole ton of sense — what with its asynchronous behavior and being an interpreted language and all.

Add to that the yearly updates the ECMAScript committee is releasing, and it’s a lot of info to keep abreast of. 🤯 And check out this statistic below from Wikipedia.

As of May 2017 94.5% of 10 million most popular web pages used JavaScript. — JavaScript, Wikipedia

Since JS powers so much of the web, I wanted to provide a bunch of articles and examples of ES6+ features that I use regularly, for other developers to reference.

For my fourth post, I’ll be discussing rest parameters and the spread syntax: the most concise ways to condense or expand arguments, elements and object values with ease.

... Rest Parameters

Before I dive into rest parameters, I recommend you familiarize yourself with the three main types of functions in use in JavaScript today: function declarations, function expressions and arrow functions. I wrote a blog post about them, for a quick refresher.

Now that we’re on the same page, let’s get into what exactly rest parameters are.

Rest parameters are a new syntax that allows us to represent an indefinite number of arguments passed to a function as an array. Make sense? Not quite? 🤔 No problem, code examples usually help me grasp concepts easier too.

Anatomy of rest parameters in a function declaration

function f (x, y, ...a) {
  return (x + y) * a.length;
}

console.log(f( 1, 2, “hello”, true, 7)) // prints: 9

When you’re looking at the example above, everything stays the same as with a normal function declaration except that the last argument declared, argument a has a suspicious ... in front of it. This is an example of a rest parameter.

What this means when the function executes is "take all remaining arguments being passed to this function, shove them all into an array, and do with them whatever the function body dictates in its statement body."

Here’s another example, but this time with an arrow function (which easily translates to a function expression in non-ES6 syntax, as well).

Rest parameters in an arrow function

const product = (e, f, ...g) => {
  return e * f + g.length; 
}

console.log(product(4, 7, 2, 6, 3)); // prints: 31

Now at this point, you may have some questions about exactly how rest parameters can be used (I know I sure did), so let me lay out the rules for you.

Rest parameter rules, traits & use cases

Only the last param can have ...

No, I didn’t lose my train of thought there, what I’m saying is: only the last parameter defined in a function, can be prefixed with the ..., making it a rest parameter.

When the last parameter is prefixed with ..., all remaining (user supplied) arguments are placed within a "standard" Javascript array.

Output of arguments with a rest parameter

Here’s the kind of output you’ll see when you have a rest parameter as the final argument in a function call. As you can see, the first three arguments passed to myNumberFunction, the 1, 2, and 3 are printed out individually as the first three parameters defined in function: a, b, and c. All the remaining numbers passed to the rest parameter d, however, are printed out as an array [4, 5, 6, 7, 8].

function myNumberFunction(a, b, c, ...d) {
  console.log("a", a);  // a 1
  console.log("b", b); // b 2
  console.log("c", c); // c 3
  console.log("...d", d); //...d [ 4, 5, 6, 7, 8]
}

myNumberFunction(1, 2, 3, 4, 5, 6, 7, 8);

Rest parameters are not just an arguments object

There are three main differences to know about between the arguments object and new rest parameters.

Rest parameters are only the ones that haven’t been given a separate name (i.e. formally defined in function expression), while the arguments object contains all arguments passed to the function.
The arguments object is not a real array, while rest parameters are Array instances, meaning methods like sort(), map(), forEach(), or pop() can be applied on it directly.
The arguments object has additional functionality specific to itself (like the callee property).

Rest parameters were originally introduced to reduce the boilerplate code caused by arguments. Before, it was a major pain to convert arguments to a "normal array". Now, they can simply be passed in and then actioned upon.

Easy actioning on rest parameter arrays

function doubleUp(...simpleArgs){
  const arr = simpleArgs;
  const secondArr = arr.map(num => num * 2);
  return secondArr;
}

console.log(doubleUp(2, 6, 12, 18)); // [ 4, 12, 24, 36 ]

Rest parameters can be destructured

One of my favorite new things about ES6 is array and object destructuring. I haven’t gone into detail about destructuring yet, but just wait, I will, before this series of articles is finished.

In a nutshell, destructuring makes it possible to unpack values from arrays, or properties from objects, into distinct variables. If you’re dying to learn more in the meantime, here’s a link to get more familiar with how destructuring works.

Moving on for the purposes of this post though, here’s rest parameters and destructuring at work.

Array destructuring with a rest parameter

const [captain, coCaptain, ...devs] = [ "Bridget", "Joe", "Kyle", "Drew", "Patrick", "Francisco"];
console.log(captain); // "Bridget"
console.log(coCaptain); // "Joe"
console.log(devs); // ["Kyle", "Drew", "Patrick", "Francisco"]

In the above declared array of variables, captain, coCaptain and devs, the rest parameter is applied to devs, so when the array of names is passed in, "Bridget" becomes the captain, "Joe" is the coCaptain and the devs are ["Kyle", "Drew", "Patrick", "Francisco"].

Handy, right? Just imagine the scenarios you’ve encountered where you either needed to pull out individual pieces of an array or keep other pieces together. Trust me, you’ll be so thankful for destructuring and rest params when you do.

The same type of thing can be done with object destructuring too.

Object destructuring with a rest parameter

const {pm1, pm2, ...restOfTeam} = {
  pm1: "Jeremy",
  pm2: "Tung",
  developer1: "Casey",
  developer2: "Mark",
  ux: "Christina"
};

console.log(pm1); // Jeremy
console.log(pm2); // Tung
console.log(restOfTeam); // { developer1: "Casey", developer2: "Mark", ux: "Christina" }

This example deconstructs pm1 and pm2 from the rest of the team members in the object, and the remaining object properties: developer1, developer2, and ux are grouped together in a new object variable called restOfTeam.

Again, think about using this type of syntax to create new variables, objects, pull object properties apart into individual pieces, etc.

And those are the main things you need to know about rest parameters. At this point we can move on to the other use of ... in JavaScript, the spread syntax.

Spread syntax

While it resembles the rest parameter a great deal, the spread syntax has quite different uses.

The spread syntax allows an iterable such as an array or string to be expanded in places where zero or more arguments (for function calls) or elements (for array literals) are expected, or an object expression to be expanded in places where zero or more key-value pairs (for object literals) are expected.

Uh... What now?

Ok, that definition sounds technical, I know, believe me. But here’s what it means in practice:

If you’ve got a string, an array or an object, and you want to use all the values, you can spread all of them into function calls, new arrays, or new objects, with a very concise syntax.

The spread syntax at work in functions, arrays, strings, and objects

// For function calls:
function multiply(x, y, z) {
  return x * y * z;
}

const args = [1, 2, 3];

console.log(multiply(...args)); // 6

// For array literals or strings:
const iterableObj = [ {protein: "steak"}, {carb: "potato"}, "milkshake"];
const randomList = [...iterableObj, "4", "five", 6];
console.log(randomList); // [ { protein: "steak" }, { carb: "potato" }, "milkshake", "4", "five", 6 ]
const str = "foo"
const chars = [ ...str ] 
console.log(chars); // [ "f", "o", "o" ]

// For object literals (new in ECMAScript 2018):
const powerTool = { skuNumber: "996655", skuDescription: "Drill Bit" };
let secondPowerTool = { ...powerTool, toolDepartment: 25, toolClass: 7 };
console.log(secondPowerTool); // { skuNumber: "996655", skuDescription: "Drill Bit", toolDepartment: 25, toolClass: 7 }

The different examples above show what happens when you spread various items.

For the first example, the multiply() function, I spread my args array into the function call, which simply takes the three values from the array (1, 2, and 3) and puts them in the places of the function’s accepted parameters x, y, and z.

In the case of the second and third examples, spreading arrays and strings, the original list, iterableObj, and the original string, "foo", are both spread into new variables, randomList and chars, respectively.

All the values in iterableObj are added to the new randomList array along with the new values assigned specifically to randomList. And the original string of "foo" is spread apart, character by character, into the new chars array.

The final example, only possible since the release of ECMAScript 2018, is of spreading the object powerTool's properties into a new object called secondPowerTool, and adding two new properties to that object as well: toolDepartment and toolClass.

Spread syntax rules, traits & use cases

Just like the rest parameter, the spread syntax is syntactic sugar that replaces complex methods and boilerplate code-filled ways of doing things that should be relatively easy.

That being said, there are things that you need to be aware of regarding spread.

Spread Replaces Apply in Function Calls

Up to now, it’s been common to use Function.prototype.apply() in cases where you want to use the elements of an array as arguments to a function.

Using apply() to assign array elements in a function call

function myFunction(x, y, z) { }
var args = [0, 1, 2];
myFunction.apply(null, args);

With spread, the syntax is much cleaner.

Spreading a list of array elements into a function call

function myFunction(x, y, z) { }
var args = [0, 1, 2];
myFunction(...args);

And something else that’s cool: any argument in the argument list can use spread syntax and it can be used multiple times.

Multiple spreads in a single function call

function myFunction(v, w, x, y, z) { }
var args = [0, 1];
myFunction(-1, ...args, 2, ...[3]);

Spread syntax can be used with the new keyword

Unlike when calling a constructor with new, where it's not possible to directly use an array and apply, with spread, an array can be easily used with new.

Spread syntax with the new constructor

var dateFields = [1989, 3, 13];
var d = new Date(...dateFields);
console.log(d); // 13 Apr 1989

Spread in array literals

Spread makes array manipulation so much easier, and methods like push(), concat(), and splice() a lot less necessary. Let’s go over what it can help you do.

Creating new array literals is a snap

Create a new array using an existing array as one part of it requires no additional array methods like it would have in the past.

Spreading one array into another new array

const fruits = ["watermelon", "peaches"];
const fruitBasket = ["apples", "grapes", ...fruits, "bananas", "kiwis", "mango"];
console.log(fruitBasket); // [ "apples", "grapes", "watermelon", "peaches", "bananas", "kiwis", "mango" ]

How easy is that to add the fruits array to fruitsBasket? Super easy.

Copying arrays is a breeze too

With spread, copying the values of one array into another array is a piece of cake. Then you can continue to modify the newly copied array without affecting the original array (which lends itself well to the style of immutable, functional programming so popular today with frameworks like React and state management tools like Redux).

const arr1 = [1, 2, 3];

const arr2 = [...arr1];

console.log(arr2); // [1, 2, 3]

arr2.push(4);

console.log(arr1); // [1, 2, 3]

console.log(arr2); // [1, 2, 3, 4]

Before I pushed 4 onto arr2, when the values were printed out, they were exactly the same as arr1. And once again, even after 4 is added to arr2, arr1 remains unaltered.

NOTE: Spread syntax only effectively goes one level deep while copying an array. Therefore, it may be unsuitable for copying multidimensional arrays, for that using a function like lodash’s _.cloneDeep() is recommended.

Concatenating arrays has never been easier

Array.prototype.concat() was the go-to when concatenating an array onto the end of an existing array. Without spread syntax this is done as:

.concat() Arrays Together

const germanCars = [ "BMW", "Audi", "Mercedes" ];
const japaneseCars = [ "Honda", "Toyota", "Datsun" ];

const concatCarMakers = germanCars.concat(japaneseCars);

console.log(concatCarMakers); // [ "BMW", "Audi", "Mercedes", "Honda", "Toyota", "Datsun" ]

With spread syntax this becomes:

Spreading arrays together

const germanCars = [ "BMW", "Audi", "Mercedes" ];
const japaneseCars = [ "Honda", "Toyota", "Datsun" ];

const carMakers = [...germanCars, ...japaneseCars];

console.log(carMakers); // [ "BMW", "Audi", "Mercedes", "Honda", "Toyota", "Datsun" ]

Spread syntax can also take the place of unshift() and make adding new elements to the front of an array much simpler than it used to be.

Spread elements into the beginning of an array

let numbers = [ 6, 5, 4 ];
const moreNumbers = [ 1, 2, 3 ];

numbers = [...moreNumbers, ...numbers];

console.log(numbers); // [ 1, 2, 3, 6, 5, 4 ]

Spread in object literals

As of ECMAScript 2018, spread properties came to object literals. This lets an object copy its own enumerable properties from a provided object onto a new object.

Shallow cloning and merging objects is simple

While Object.assign() made shallow-cloning and merging objects together possible, using the spread syntax is even more concise.

// duplicate object properties
const markerSet = { copicMarkers: ["green", "blue", "red"]};
const duplicateMarkerSet = {...markerSet};

console.log(duplicateMarkerSet); // { copicMarkers: [ "green", "blue", "red" ] }

// merge two objects into a new one
const markerSet2 = { copicSketchMarkers: ["pink", "yellow", "orange"]};
const giantMarkerSet = { ...markerSet, ...markerSet2 };

console.log(giantMarkerSet); // { copicMarkers: [ "green", "blue", "red" ], copicSketchMarkers: [ "pink", "yellow", "orange" ] }

It’s also worth noting that Object.assign() triggers setters, which binds an object property to a function to be called when there is an attempt to set that property, whereas the spread syntax does not.

Spread (aside from object properties) is only for iterables

The spread syntax (other than in the case of spread properties) can be applied only to iterable objects: Arrays, Maps, Sets, Strings and the like.

// spreading in an object to an array does NOT work
const obj = { key1: "value1"};
let array = [...obj];

console.log(array); // []

// spreading an array of objects into another array does work
const objInArray = [ { key2: "value2"}];
array = [...objInArray];

console.log(array); // [ { key2: "value2" } ]

And that’s about the long and short of the spread syntax details that you need to know to use it effectively.

Conclusion

At first glance, some of the newest JS syntax can seem totally foreign — even to people who’ve been writing JavaScript code for years. And while yes, it is different, it’s also incredibly powerful and makes doing our jobs so much easier than they were even just a few years ago.

My aim with this blog series is to explain some of the JavaScript and ES6 syntax you use everyday, and show you how to fully harness the newest parts of the JavaScript language to full effect.

Rest syntax looks exactly like spread syntax (...) but is used for destructuring arrays and objects. In a way, rest syntax is the opposite of spread syntax: spread expands an array into its elements, while rest collects multiple elements and condenses them into a single element. It’s so useful in so many common programming situations, as I’m sure you can already imagine.

Check back in a few weeks, I’ll be writing about more JavaScript and ES6 or something else related to web development.

Thanks for reading, I hope you’ll start seeing the possibilities of using rest parameters and the spread syntax in your code in the future — they make a myriad of things a cinch. Please share this with your friends if you found it helpful!

References & Further Resources

Rest Parameters, MDN Docs
Spread Syntax, MDN Docs
JavaScript, Wikipedia
Destructuring, MDN Docs

Default Function Parameter Values: JavaScript ES6 Feature Series (Pt 3)

Fri, 23 Aug 2019 00:00:00 GMT

</figure>

Introduction

The inspiration behind these posts is simple: there are still plenty of developers for whom JavaScript makes no sense — or at the very least, is bewildering at times.

Since JS does power so much of the web, I wanted to provide a slew of articles about JavaScript ES6 features that I use regularly, for other developers to reference.

For my third post, I wanted to talk about default function parameter values: the things that will save us all from tons of undefined checks in our code.

Traditional function parameters

As one of my previous posts detailed, functions are one of the fundamental building blocks of JavaScript. One of the things functions allow us to do is define a set of statements that performs a task or calculates a value the same way, every time, AND supply a list of parameters that can be passed in to be operated on.

Regardless of what values are passed in as parameters, the function will attempt to run the same tasks every time it is called.

Anatomy of a function declaration and parameters

function simpleAdd(a, b){
  return a + b;
}

simpleAdd(2, 3); 
// prints: 5

If you’re looking at the function declaration above, here’s what composes it. simpleAdd is the name of the function, a and b are the two parameters the function takes in, and the body of the function return a + b; is the statement.

Ok, so that’s simple, straightforward and makes perfect sense, right? Right. But what happens if a traditional function declaration isn’t passed the correct amount of parameters?

An overabundance of parameters is OK, the JavaScript engine just knows to ignore additional arguments that it doesn’t use in the function, but not supplying enough parameters is a different problem entirely... Check out what happens there.

Function declaration with too many & too few parameters

function add(a, b){
  return a + b;
}

console.log(add(1, 2, 3, 4)); 
// prints: 3

console.log(add(1)); 
// prints: NaN

Uh-oh, the lack of a b parameter didn’t turn out so well for the add(1) function. NaN ("not a number") is something a developer never wants to see when attempting to do math with JavaScript.

Why did this happen? Because traditionally, function parameters default to undefined when the function is declared, and it waits for arguments to be passed in when the function is called in the code. So, how do we avoid dreaded things like NaN? Up until ES6, we had to do things like undefined checks everywhere in functions.

Checking a function for undefined parameters

function addImproved(a, b) {
  if(a === undefined){
    a = 3;
  }
  if(b === undefined){
    b = 7;
  }
  return a + b;
}

console.log(addImproved(6,9)); 
// prints: 15 

console.log(addImproved()); 
// prints: 10

In the example above addImproved(), in order to avoid undefined parameter errors, we have to physically check each parameter as it’s passed in and make sure that it is not undefined before finally letting the function get to its main goal: adding the two numbers together.

What a pain — imagine having to write this same type of logic over and over again for hundreds of functions throughout a large code base. 😩 No thanks, there’s got to be a better way...

Default function parameters

Enter the default function parameters; they allow named parameters in a function to be initialized with default values if no value or undefined is passed in.

As I briefly explained above, up until now, in JavaScript, function parameters automatically default to undefined; which meant we had to test parameter values in a function body, check for potentially undefined parameters and assign values if they were undefined.

Here’s an example of two ways to set values if they aren’t assigned. The first is a simple if / else statement, the second is a ternary statement checking if the parameter’s type is undefined or not, and setting the value if need be.

Two traditional ways of preventing undefined parameters

function oldSum (x, y, z) {
  if (y === undefined) {
    y = 7;
  }
  z = (typeof z !== "undefined") ? z : 42;
  return `oldSum(${x}) === ${x + y + z}`;
};

console.log(oldSum(1));  
// prints: oldSum(1) === 50

console.log(oldSum(2, 5, 8)); 
// prints: oldSum(2) === 15

console.log(oldSum()); 
// prints: oldSum(undefined) === NaN

Seems like a lot of extra code just to prevent errors before the function even runs. With default parameters in ES2015, checks in the function body are no longer necessary.

ES6 ways to prevent undefined errors with default parameters

function newSum (x, y = 7, z = 42) {
  return `newSum(${x}) === ${x + y + z}`;
};

console.log(newSum(1)); 
// prints: newSum(1) === 50

console.log(newSum(3, 6, 9)); 
// prints: newSum(3) === 18

console.log(newSum(16, undefined)); 
// prints: newSum(16) === 65

Much nicer! Right where the parameters are named, they are also given default values in case not enough arguments are supplied. It’s so much cleaner and easier to read this way, and removes the boilerplate of undefined checks.

I should also mention, the same rules that apply to default parameters in function declarations, work in arrow functions, as well. Check this out.

ES6 default parameters in arrow functions

const thisWayWorksToo = (x = 7, y = 8) => {
  return x + y;
}

console.log(thisWayWorksToo(3, 4)); 
// prints: 7

console.log(thisWayWorksToo()); 
// prints: 15

These types of examples are far and away the most common use cases for default parameters, however, there’s more to know about them, if you’re curious. 🤔

Gotchas and other use cases for default parameters

Default params have some other interesting functionalities it will be helpful to know — they also have some really handy use cases. Let’s go over them so you don’t get tripped up, or when you do, you can more quickly debug the source of the problem.

Passing undefined vs other falsy values

One thing that could trip you up: passing other falsy values like null or '’ to a function with default parameters will not result in the default value being substituted instead.

function luckyNumber(num = 11) { 
  console.log(typeof num);
};

luckyNumber(); 
// prints: 'number' (num is set to 11)

luckyNumber(undefined); 
// prints: 'number' (num is set to 11 too)

// test with other falsy values:
luckyNumber(''); 
// prints: 'string' (num is set to '')

luckyNumber(null); 
// prints: 'object' (num is set to null)

As you can see above when luckyNumber() is called with no value or a value of undefined it falls back to its default param of 11. When it’s called, however, with an empty string or a null value, it takes that value instead.

Evaluation at call time

The second thing to know is: default values are evaluated at call time, so a new object is created each time the function is called — not, contrary to what you might think, adding to an already existing object or array.

function append(value, array = []) {
  array.push(value);
  return array;
}

console.log(append(1)); 
// prints: [1]

console.log(append(2)); 
// prints: [2], not [1, 2]

Each time the append() function runs, it creates a brand new array with the value supplied as its contents. The first time the function is called and executed it runs, makes an array and is done, and its execution context is destroyed once it’s done running.

What this means is that the function is now completely done with (and unaware of) the original array it created. Which is why, when append() is invoked a second time, with a second value, it creates a second, entirely new array, instead of adding to the first one it created.

Earlier params are available to later default params

Ok, here’s a cool trick: parameters declared earlier (to the left) in a parameter string are then available to be used by later default parameters.

function welcome(name, greeting, message = greeting + ' ' + name) {
  return [name, greeting, message];
}

console.log(welcome('Sean', 'Hi'));  
// ["Sean", "Hi", "Hi Sean"]

console.log(welcome('Sean', 'Hi', 'Happy Birthday!')); 
// ["Sean", "Hi", "Happy Birthday!"]

By passing in values for the name and greeting parameters, the third param of message is able to take those two to make another string value. Pretty sweet.

Imagine how much more cleanly you could handle functions with lots of potential variables. This example, which I borrowed from the MDN docs on default parameters really drives the point home.

Cleaner edge case handling with and without default parameters

function go() {
  return ":P";
}

function withDefaults(a, b = 5, c = b, d = go(), e = this, f = arguments, g = this.value) {
  return [a, b, c, d, e, f, g];
}

function withoutDefaults(a, b, c, d, e, f, g) {
  switch (arguments.length) {
    case 0:
      a;
    case 1:
      b = 5;
    case 2:
      c = b;
    case 3:
      d = go();
    case 4:
      e = this;
    case 5:
      f = arguments;
    case 6:
      g = this.value;
    default:
  }
  return [a, b, c, d, e, f, g];
}

withDefaults.call({value: "=^_^="});
// [undefined, 5, 5, ":P", {value:"=^_^="}, arguments, "=^_^="]

withoutDefaults.call({value: "=^_^="});
// [undefined, 5, 5, ":P", {value:"=^_^="}, arguments, "=^_^="]

Yeah, is there anyone who would choose to write out the withoutDefaults() function instead of the withDefaults() function? I think not.

Parameters without defaults after default parameters

While earlier params are available to later default params, you cannot, pass in defined parameters to fill the place of later undefined parameters in a function. Makes perfect sense, right? 😉 To put it another way: parameters are always set left-to-right, and they overwrite default parameters even if there are later parameters without defaults. This example below probably illustrates it better.

function f(x = 1, y) {
  return [x, y];
}

console.log(f()); 
// prints: [1, undefined]

console.log(f(4)); 
// prints: [4, undefined]

And those are some of the main things to know when it comes to default parameters in functions.

Conclusion

Even though JS has been around for more than 20 years and ES6 has been out since 2015, there’s still plenty of misinformation and knowledge gaps surrounding it.

I’m trying to demystify it and give you a better understanding of JavaScript, ES6 — things you may use everyday but not fully grasp the nuances of.

Functions are one of the key building blocks of JavaScript, and before now, checking for undefined parameters was just a fact of life. With the introduction of default parameters, though, at least some of the boilerplate can be peeled away, making functions easier to write, read and maintain, and I, for one, am incredibly grateful, and excited for future improvements as well.

Check back in a few weeks, I’ll be writing about more JavaScript and ES6 or something else related to web development.

Thanks for reading, I hope you’ll have a chance to take advantage of default parameters in the functions you write going forward, be they declarations, expressions or arrow functions. Please share this with your friends if you found it helpful!

References & Further Resources

Arrow Functions: JavaScript ES6 Feature Series (Pt 2)

Tue, 06 Aug 2019 00:00:00 GMT

</figure>

Introduction

The inspiration behind this series of posts is simple: there are still plenty of developers for whom JavaScript makes no sense sometimes — or at the very least, has seemingly odd behavior compared to other programming languages.

Since it is such a popular and widely used language though, I wanted to provide a bunch of posts about JavaScript ES6 features that I use regularly, for developers to reference.

The aim is for these articles to be short but still in-depth explanations about various improvements to the language, that I hope will inspire you to write some really cool stuff using JS. Who knows, you might even learn something new along the way. 😄

For the second post in this series, I wanted to dive into arrow functions, and how they differ from traditional function declarations and function expressions.

Function declarations

You may have heard this before, but it bears repeating: in JavaScript, functions are first-class objects, because they can have properties and methods just like any other object. What distinguishes them from other objects is that functions can be called. In brief, they are Function objects.

From here on, I will assume you’re familiar with the general idea of functions in JavaScript, But before I talk about arrow functions, it’s worth talking a little about both function declarations (also known as function statements) and function expressions.

Function declarations are the most basic function statement we see all the time in JavaScript. It defines a function along with the specified parameters it needs to run.

Here’s an example:

Anatomy of a function declaration

function multiply(number1, number2){
  return number1 * number2;
}

console.log(multiply(4, 9));  
// prints 36 to the console

If you’re looking at the function declaration example above, here’s what composes the function. multiply is the name of the function, number1 and number2 are the two parameters the function takes in, and the body of the function: return number1 * number2; is the statement.

Function declaration traits

Function declarations have certain traits which developers need to keep in mind as they write code, because they will trip you up at one time or another — they manage to trip us all up (myself included). 🙋

1. Functions return undefined, unless otherwise specified

By default functions return undefined. By including the return keyword in the body, you can specify the value it returns instead.

Function declarations undefined vs. returned values

function returnsNothing(item1, item2) {
  item1 + item2;
}

console.log(returnsNothing(1, 9)); 
// prints: undefined

function returnsSomething(item1, item2) {
  return item1 + item2;
}

console.log(returnsSomething(1, 9)); 
// prints: 10

For my example above, the sum of item1 and item2 is what is returned from the returnsSomething() function, while the returnsNothing() function, although it does exactly the same addition, merely returns undefined when the value is called with console.log().

2. Function declarations are hoisted

Similar to variable hoisting, which I discussed in my previous blog post, function declarations in JavaScript are hoisted to the top of the enclosing function or global scope. This means, you can use a function before it’s actually been declared in the code.

Function declaration hoisting vs. function expression not hoisting

console.log(hoistedFunction()); 
// prints: "Hello, I work even though I am called before being declared"

function hoistedFunction() {
  return "Hello, I work even though I am called before being declared";
}

console.log(notHoisted());
 // prints: TypeError: notHoisted is not a function

var notHoisted = function() {
  return "I am not hoisted, so I will not be found if called before my declaration";
}

console.log(notHoisted()); 
// prints: "I am not hoisted, so I will not be found if called before my declaration"

In the above example, the function hoistedFunction() returns its value, regardless of when it’s called in the code, because it’s a function declaration.

On the other hand, the second function assigned to the variable notHoisted(), which is a function expression, is not hoisted to the top of the scope so if it is invoked before the function is parsed, it throws a TypeError in the code that it is not a function (mainly because the compiler’s not aware of it yet).

Those are main things you need to be aware of when thinking about function declarations.

Let’s move on to function expressions.

Function expressions

Function expressions are similar to function declarations. They still have names (which are optional this time), parameters, and body-based statements.

Anatomy of a function expression

const divide = function(number1, number2){
  return number1 / number2;
}

console.log(divide(15, 5)); 
// prints: 3

For this example, the variable divide() is assigned to the anonymous function which takes the parameters number1 and number2, and returns the quotient as per the body’s statement return number1 / number2;.

Function expression traits

Just like function declarations, function expressions have some defining quirks of their own. Here’s what you need to know about them.

1. Function expressions can be anonymous (or not)

As I mentioned briefly above, function expressions, since they’re assigned to variables, can omit a name, and be what’s known as an "anonymous function". This is possible because the variable name will be implicitly assigned to the function.

Anonymous function expression (implicit naming at work)

const anonymous = function() {
  return "I do not need my own name, as I am assigned to the variable anonymous";
}

console.log(anonymous()); 
// prints: "I do not need my own name, as I am assigned to the variable anonymous"

As the variable name implies, since the function it’s referencing has no name, it implicitly assigns anonymous() to the function.

If, however, you want to refer to the current function inside the function body, you need to create an explicitly named function (whose name is local only to the function body).

Named function expression (explicit naming at work)

var math = {
  "factit": function factorial(n) {
    console.log(n)
    if (n <= 1) {
      return 1;
    }
    return n * factorial(n - 1);
  }
};

console.log(math.factit(3)); 
//prints: 3; 2; 1;

For this variable math, you can call the factorial() function by invoking math.factit(); outside of the object, and passing the required parameter in.

I don’t find as much need for this type of named function expression in my day to day development, but it’s nice to know it’s available if the need arises.

Bottom line: If the function expression’s name is omitted, it will be given the variable name (implicit name). If the function expression’s name is present, it will be the assigned function name (explicit name).

2. Function expressions can be IIFEs

A function expression can be used as an IIFE: an Immediately Invoked Functional Expression, which runs as soon as it’s defined.

This immediate execution by the JavaScript engine is triggered by the parentheses () at the end of the anonymous function.

I won’t go into too much detail here, but for any variables created within the IIFE to be accessible to the outside or global scope, the anonymous function must be assigned a variable a la a function expression.

If it’s not and the anonymous function’s just invoked at run time, any variables created inside the function’s scope will be invisible to the outside world.

IIFE variables that are accessible vs. IIFEs that are not

const cogitoErgoSum = (function () {
  const quote = "I think therefore I am";
  return quote;
})();

// immediately creates the output
cogitoErgoSum; 
// prints: "I think therefore I am"

(function (){
  const quote2 = "I am not outside this IIFE";
})();

quote2; 
// prints: ReferenceError: quote2 is not defined

3. Function expressions don't hoist

Function expressions (and arrow functions) take after the new let and const variable keywords, in that they don’t get hoisted at run time.

As I demonstrated above in the function declaration section around hoisting, function expressions do not hoist, a function expression is created when the execution reaches it and it is usable from then on.

Function declaration hoisting vs. function expression not hoisting

console.log(hoistedFunction()); 
// prints: "I am a function declaration so I am hoisted to the top of the scope at run time"

function hoistedFunction() {
  return "I am a function declaration so I am hoisted to the top of the scope at run time";
}

console.log(stillNotHoisted()); 
// prints: TypeError: stillNotHoisted is not a function

const stillNotHoisted = function() {
  return "I am a function expression and therefore, hoisting does not apply to me";
}

console.log(stillNotHoisted());
 // prints: "I am a function expression and therefore, hoisting does not apply to me"

The outcome is the same as when I described it in function declarations, function expressions throw TypeErrors if invoked before they are parsed at run time. Just don’t do it.

Ok, now time to move on to arrow functions: the latest and greatest function improvements courtesy of ES6.

Arrow function expressions ➡️

<figcaption>The most basic arrow function syntax.</figcaption>

</figure>

Arrow function expressions are syntactically compact alternatives to regular function expressions.

Anatomy of two basic arrow function expressions

const basicArrow = () => {
  return "The most basic of basic arrow functions";
}

basicArrow(); 
// prints: "The most basic of basic arrow functions"

const basicArrow2 = oneParam => `Single line with ${oneParam} is also valid`;

basicArrow2("only one param"); 
// prints: "Single line with only on param is also valid"

Both of the examples above basicArrow() and basicArrow2() are valid examples of an arrow function. As with all function expressions, both anonymous functions are implicitly named by the variables assigned to them.

What’s different though is that the function keyword is unnecessary. Instead it’s replaced by a set of parentheses () if there are no required parameters, the name of the single parameter required by basicArrow2(), which is oneParam (no parentheses necessary in this case), or, for any other number of parameters, you could do (paramOne, paramTwo, paramThree, ...).

Similarly, the first function has a normal return statement inside the body of the function surrounded by curly braces {} but, if the statement is super simple, and you can fit the return on a single line, the actual return and curly braces can also be omitted, like in basicArrow2(). This is concise body syntax with an implied return statement.

Arrow function traits

While arrow functions are easy to identify at first glance, they actually have some traits which are odd and specific to them, and which developers need to keep in mind.

In addition to their concise syntax, arrow functions lack the this, arguments, super, or new.target keywords.

These facts also lend themselves to one of the arrow function’s biggest drawbacks: they are ill suited as methods and cannot be used as constructors. I’ll get to that in more detail soon.

1. Shorter function syntax

The first, and biggest improvement, in my opinion, to regular function expressions is the shorter, more concise syntax that arrow functions offer.

Here’s the exact same function written as a traditional function expression, then written again as an arrow function expression.

Traditional function expression

var elements = ["Hydrogen", "Helium", "Lithium", "Beryllium"];

elements.map(function(element) {
  return element.length;
});
// this statement returns the array: [8, 6, 7, 9]

New arrow function expression

var elements = ["Hydrogen", "Helium", "Lithium", "Beryllium"];

elements.map((element) => element.length);
// this statement still returns the same array: [8, 6, 7, 9]

Look at those and tell me the second function isn’t easier to read and follow what the code is doing.

That by itself, is my biggest reason to want to use arrow functions whenever possible. It’s just so much cleaner and clearer.

2. Hoisting still doesn't apply

Just like with traditional function expressions, hoisting still does not apply to arrow functions.

No hoisting, only TypeErrors

console.log(fish()); 
// prints: TypeError: fish is not a function

const fish = () => ["perch", "salmon", "trout", "bass"];

console.log(fish());
// prints: ["perch", "salmon", "trout", "bass"]

If you try to call the fish() function before it’s declared in the code, a TypeError will be thrown.

The solution, as before, is to either declare the function as a function declaration so it gets hoisted to the top of the scope, or wait until after the function expression to call the code.

3. No separate "this"

Before arrow functions, every new function defined its own this value based on how the function was called:

A new object in the case of a constructor,
undefined in strict mode function calls,
The base object if the function was called as an "object method".

An arrow function, on the other hand, does not have its own this.

The this value of the enclosing lexical scope is used. Arrow functions follow the normal variable lookup rules of starting at the current scope level and searching all the way to the highest level looking for the variable. So while searching for this which is not present in the current scope, an arrow function ends up finding the this object from its enclosing scope.

See the example below to see the difference.

this scope, according to function declarations

function Person() {
  // The Person() constructor defines `this` as an instance of itself.
  this.age = 0;

  setInterval(function growUp() {
    // In non-strict mode, the growUp() function defines `this`
    // as the global object (because it's where growUp() is executed.), 
    // which is different from the `this`
    // defined by the Person() constructor.
    this.age++;
  }, 1000);
}

var p = new Person();

this scope, according to arrow functions

function Person(){
  this.age = 0;

  setInterval(() => {
    // `this` properly refers to the Person object
    this.age++; 
  }, 1000);
}

var p = new Person();

4. No binding of arguments

In addition to not having access to this, arrow functions do not have their own arguments object.

Since you may not have heard of them, arguments is an Array-like object inside a function that contains the values passed to that function. I say Array-like because this arguments has a length property and indexing but it lacks Array’s built-in methods like .forEach() and .map().

Thus, in this example, arguments is simply a reference to the arguments of the enclosing scope.

arguments with arrow functions

var arguments = [1, 2, 3];
var arr = () => arguments[0];

arr(); 
// prints: 1

function foo(n) {
  var f = () => arguments[0] + n; 
  // foo's implicit arguments binding. arguments[0] is n
  return f();
}

foo(3); 
// prints: 6

In most cases, using rest parameters is a good alternative to using an arguments object.

Rest parameters instead of arguments with arrow functions

function foo(n) { 
  var f = (...args) => args[0] + n;
  return f(10); 
}

foo(1); 
// prints: 11

The rest parameters, which I’ll discuss in another blog post later in this series, is the ES6-recommended way of accessing and manipulating arguments inside of arrow functions.

5. No use of "new" as a constructor

Ok, last arrow function trait to know about: arrow functions cannot be used as constructors and will throw a TypeError when used with the new keyword.

var Foo = () => {};

var foo = new Foo(); 
// prints: TypeError: Foo is not a constructor

And that’s it. That’s generally what you need to know about arrow functions. Simple! 😅

Conclusion

JavaScript is an incredibly powerful programming language, and its popularity only continues to increase (if the yearly developer surveys are to be believed). With so many developers using it though, there’s bound to be some misconceptions and knowledge gaps, especially with the growing widespread adoption of ES6 into everyday use.

My aim is to shed light on some of the JavaScript and ES6 syntax you use everyday, but may never have fully grasped the nuances of why it works the way it works.

Functions of all kinds are a staple of JavaScript, be they traditional function declarations or function expressions, or the newer, more concise ES6 arrow functions. Knowing when (and how to) effectively take advantage of the benefits of each type of function will definitely make writing JS easier.

Check back in a few weeks, I’ll be writing about JavaScript, ES6 or something else related to web development.

Thanks for reading, I hope you feel better equipped to incorporate arrow functions into your JavaScript applications. Please share this with your friends if you found it helpful!

References & Further Resources

Var, Let & Const: JavaScript ES6 Feature Series (Pt 1)

Sun, 28 Jul 2019 00:00:00 GMT

</figure>

Introduction

The inspiration behind these posts is simple: there are still plenty of developers for whom JavaScript is a mystery — or at the very least, a little less understood.

Add to that ES6, and the updates that the ECMAScript committee is now planning to release yearly, and it’s a lot of info to keep abreast of.

<figcaption>Sorry, couldn’t help myself — I love Barney from How I Met Your Mother.</figcaption>

</figure>

I wanted to provide a bunch of posts about JavaScript ES6 features that I use regularly, for developers to reference.

The aim is for these articles to be short, in-depth explanations about various improvements to the language (and how to use them), that I hope will inspire you to write some really cool stuff using JS. Who knows, you might even learn something new along the way. 😄

For my first post, it makes sense to begin at the beginning with var, let, and const: the building blocks of JavaScript.

Var

var is the original JavaScript syntax: the OG way to declare a variable, and optionally, initialize it as a value. It’s been around since the beginning of the language, long before ES6 was even a thought in anyone’s head.

Variables have no pre-defined or attached type in JavaScript, meaning any value can be stored in any variable. Some examples:

Assigning values to variables

var one = 1; // the integer 1 has just been assigned to the variable one

var two = 'two'; // the string 'two' has been assigned to the variable two

var paige = {
  gender: female,
  hairColor: red,
  eyeColor: blue
};
// the object containing a person's traits has been assigned to the variable paige

All of the vars above are valid examples of how JavaScript allows any value to be assigned to any variable, unlike strongly typed languages like Java, which (until recently), required all variables be declared and typed before compile time.

Another thing to know about var is that it is regardless of if it is globally or locally scoped, these variables can be re-declared and updated at will. Check out this perfectly valid example.

Reusing the same variable with different value types

var greet = "Hello there";
console.log(greet); // prints: "Hello there"

greet = true
console.log(greet); // prints: true

greet = 11;
console.log(greet); // prints: 11

greet = welcome = (name) => { 
  return `Hi ${name}. Welcome!`
};
console.log(greet("Paige")); // prints "Hi Paige. Welcome!"

Although I reused the existing variable greet, JavaScript throws no error, regardless of whether the variable is a string, a boolean, an integer — heck, even a function.

Variable declarations, lexical scoping, and hoisting

There are a few things to be aware of (which have tripped up many a developer, myself included), when using var. These types of variables are "lexically scoped" and subject to what’s known as "hoisting."

What this means in execution is that a var-type variable, no matter where it is defined inside of a block of code, will be pulled to the top of its scope before code execution. For example, if you wrote a few lines of JavaScript, then declared var tree = "elm";, at run time, behind the scenes, var tree would be "hoisted" to the top of the scope where it was defined.

It may not be assigned to the value of "elm" until several lines later, but the variable would exist according to the JavaScript engine. Here’s an example to show how this works, and what it could mean.

Variable declaration versus variable assignment in JavaScript

var greeter; {/* the JS engine knows greeter is a variable, 
but it doesn't know what value greeter has */}
console.log(greeter); // currently, greeter prints: undefined

greeter = "say hello"; // now, greeter's been assigned to the value "say hello"
console.log(greeter); // prints: "say hello"

Lexical scoping also comes into play here. Without going too much into the details of lexical scoping in JavaScript (because that’s a whole other blog post), the word “lexical” refers to the fact that lexical scoping uses the location where a variable is declared within the source code to determine where that variable is available.

This boils down to if a variable is created in the global scope, it’s hoisted to the top of the global scope, if the variable is created within a function, it’s hoisted to the top of the function, and so on and so forth.

Here are examples of variable hoisting and lexical scoping in action as the JavaScript engine runs.

Variable hoisting based on lexical scoping

var something; // declaration is hoisted to the top of the global scope
console.log(something); // prints: undefined

function doAnotherThing() {
  var anotherThing; {/*} declaration is hoisted to the top of the function's 
  lexical scope but not immediately assigned a value */}
  var thisOneThing = "a defined value"; // this variable is hoisted and then immediately defined

  function useAnotherThing(anotherThing) {
    console.log(`The value of another thing: ${anotherThing}`);
  }

  function useThisOneThing(thisOneThing) {
    console.log(`The value of this one thing: ${thisOneThing}`)
  }

  useAnotherThing(anotherThing); 
  {/* prints: "The value of another thing: undefined" because the variable 
  wasn't assigned a value until AFTER the function useAnotherThing() ran */}

  anotherThing = "an undefined value"; 
  // finally the variable, initialized earlier, is assigned a value

  useThisOneThing(thisOneThing); 
  {/*} prints: "The value of this one thing: a defined value" because 
  the variable was assigned a value before the function useThisOneThing() ran */}
}

something = "that thing";

console.log(something); // prints: "that thing"
console.log(doAnotherThing()); // see printouts above next to inner functions

In the example above I show how the variables something and anotherThing, by not being immediately assigned values after being declared are undefined when console.log() is called on them. Once their values are set further down the script and console.log() is called again, they print their values to the console.

The variable thisOneThing, however, is initialized and assigned a value immediately after initialization, meaning that when the function using it: useThisOneThing() is run (which is also an example of a JavaScript "closures" at work), it prints out the string "The value of this one thing: a defined value."

I think this is enough about var and how it works. There’s a million other tutorials talking about it, too. Now, let’s get on to the exciting, new ES6 variable syntax, and how these types of variables are different.

Let

let was finalized and officially introduced for widespread use with the release of ES6 in 2015. Of the two new variable types introduced, it is the much more flexible and ephemeral of the pair.

Let's similarities to var

While let is a totally new variable type, it does have share a couple of commonalities with var. For instance:

When let is initialized, it can optionally be assigned a value at initialization time,
let is also a variable type that can be mutated and / or reassigned to different data types at run-time.

And that’s pretty much where the similarities end.

Let's differences to var

Ok, here’s where it gets interesting and let begins to differ from var.

Block scoping and variable declaration and assignment

let is block scoped to the statement, or expression within which it is used, unlike var, which defines a variable globally, or locally to an entire function regardless of block scope.

Bottom line: let lives inside a chunk of code (usually) defined by curly braces like so {}.

Additionally, let’s value can be updated but not redeclared as a new variable inside of the same scope, which var-type variables can. There’s an example below showing the differences between let and var.

Valid assignments of let based on block scoping

Here is a valid example of assigning the variable greeting using the let keyword in the same file, but in different block scopes.

let greeting = "say Hi"; // one variable declared in the global scope

if (true) {
  let greeting = "say Hello instead"; 
  {/*  another variable of the same name declared within this inner, block scope */}
  console.log(greeting); // prints "say Hello instead"
}
console.log(greeting); // prints: "say Hi"

Invalid assignments of let due to block scoping

Since this let greeting variable is assigned a value in the global scope, it cannot be redeclared and reassigned to a new value within the same scope. If this scenario occurs, it throws a SyntaxError in the console.

let greeting = "say Hi";
let greeting = "say Hello instead"; //SyntaxError: Identifier "greeting" has already been declared

Doing the same type of thing with var produces no such error.

var fruit = "banana";
console.log(fruit);  // prints: "banana"

var fruit = "pear";
console.log(fruit); // prints: "pear"

Variable hoisting (or lack thereof) and global objects

Hoisting does not apply to let. Unlike var a let-type variable won’t be initialized to undefined before its definition gets evaluated by the parser.

If you try to use a let variable before the part of the code where it’s initialized, you’ll get a ReferenceError thrown.

ReferenceError of let because it was called before parsed

console.log("test 0", test)

let test; // prints: ReferenceError: test is not defined

And at the top level of programs and functions, let, unlike var, does not create a property on the global window object when declared in the top-most scope either. It’s not necessarily a bad thing, it means less pollution of the global namespace, which we can all benefit from, but it also means you can’t use the oft misunderstood this property to access said global variables either.

No global object properties with let

var x = "global";
let y = "global";

console.log(this.x); // prints: "global"
console.log(this.y); // prints: undefined

Ok, I think that’s enough about let, it’s time to move to the stricter, more immutable variable type const.

Const

const came about in the same ES6 release as let back in 2015. But where let provides increased flexibility and less permanent variables, const provides more rigidity and permanence.

It is also, by far, the most different of the three variable types.

Const's defining traits

This variable declaration shares some things with let.

const, too, is limited to block scoping,
It does not create properties on the window object when it’s created in the global scope,
When it is declared it must also have a value assigned to it,
And it is not hoisted at run time.

Block Scoping

Const has the same type of block scoping that let does, but once const is used to declare a variable, that value can’t be changed through reassignment or redeclared within that scope.

const reassignment TypeErrors and redeclaration SyntaxErrors within the same scope

const pet = "dog";
console.log(pet);  // prints: "dog"

pet = "cat"; // TypeError: Assignment to constant variable
console.log(pet);

const person = "Sean";
const person = "George"; // SyntaxError: Identifier "person" has already been declared

Both pointing the existing pet variable towards a different value or trying to redeclare person within the same scope results in errors in the console.

But with block scoping in mind, the following examples work with no issues.

Valid block scoping of variables using const

const pet = "dog";
console.log(pet);  // prints: "dog"

if (true) {
  const pet = "cat"; // TypeError: Assignment to constant variable
  console.log(pet);  // prints: "cat"
}

const person = "Sean";
console.log(person); // prints: "Sean"

function meet() { 
  const person = "George"; 
  console.log(`Meet ${person}`);
}
console.log(meet()); // prints: "Meet George"

Immutability (sort of)

const creates a read-only reference to a value. The caveat to that statement though is the value itself is mutable, but const, once assigned, cannot be reassigned.

If const is assigned to an object, the properties within that object can still be updated and changed.

const’s reference to an updatable object

const greeting = {
  message: "Hello there",
  person: "Paige"
};
console.log(greeting); // prints: { message: "Hello there", person: "Paige" }

greeting.message = "Hi there";
console.log(greeting); // prints: { message: "Hi there", person: "Paige" }

Simultaneous declaration, initialization and no global object

Global constants like const do not become properties of the window object, unlike var variables. An initializer for a constant is also required; that is, you must specify its value in the same statement in which it's declared (which makes sense, given that it can't be changed later).

const not polluting the global scope with object properties

var x = "global";

const y = "global";

console.log(this.x); // prints: "global"

console.log(this.y); // prints: undefined

If const is not assigned a value when it is initialized it will throw a SyntaxError. Likewise, if a script tries to invoke it before the parser has reached the variable, it will throw a ReferenceError just like let because const is not subject to hoisting and it cannot be set to undefined before being assigned a value.

Errors From calling variables before declaration or initializing with value assignment

console.log(fruitTree); // ReferenceError: fruitTree is not defined

const fruitTree = "fig tree";

const tree; // SyntaxError: Missing initializer in const declaration

And that is about all there is to know about const, as well.

Conclusion

The JavaScript language has been around for more than 20 years and even ES6 has been out since 2015, but even so, there’s still plenty of misconceptions and knowledge gaps about it.

My aim of this series is to dispel those myths and give you a better understanding of JavaScript, and introduce some of the ES6 syntax you may use everyday but not fully grasp the nuances of.

Since everything in JavaScript is built around variables, it seemed like an appropriate place to start with var, let, and const.

Check back in a few weeks, I’ll be writing about JavaScript, ES6 or something else related to web development.

Thanks for reading, I hope you learned something new about the most basic building blocks of JS: the humble variables. Please share this with your friends if you found it helpful!

References & Further Resources

JavaScript International Methods

Sun, 07 Jul 2019 00:00:00 GMT

</figure>

Introduction

I'm lucky in that the company I work for is based mainly in the US, with a few additional locations in Mexico and Canada. This makes my job so much easier because I don't have to worry much about internationalization: not about converting to different date formats, not about the idiosyncrasies resulting from language translations, and certainly not about the oddities regarding lists and currencies in other countries (really — who puts commas after periods when it comes to money??).

Just because these aren't my problems though, doesn't mean they aren't incredibly painful for other engineers. As an example, a lack of international standardizations (and the inability to convert numbers) resulted in a mismatch of metric units versus English units, causing NASA's Mars Orbiter to completely miss entering orbit around Mars in 1999, to the tune of $125 million.

</figure>

While the sums of money my software development team works with aren't quite so large, and our errors result in bugs in the code we can usually fix in a matter of hours or days, I can definitely empathize with the NASA engineers, and I cringe thinking about how one simple mistake like not double checking everyone was using the same measurements in Orbiter's code caused such a loss.

Which is why, when I heard the ECMAScript Internationalization API had recently been released, I was excited to see what kinds of improvements were about to make JavaScript around the world better and more standardized for everyone using it.

Today, let's check out some of the best parts of Intl: the ECMAScript Internationalization API, which provides language sensitive string comparison, number formatting, and date and time formatting.

International date time highlights

As per usual, Mozilla's MDN Web Docs, give the best summation of the new international API, and the properties that come with it:

The INTL object provides access to several constructors as well as functionality common to the internationalization constructors and other language sensitive functions. — MDN Web Docs

There's six properties that come along with the new Intl object, but in my opinion, only five of them deserve their own place in this blog post.

Intl.DateTimeFormat
Intl.RelativeTimeFormat
Intl.ListFormat
Intl.NumberFormat
Intl.PluralRules

So without further preamble, I'll dive into the first property: the international date time format.

International date time format

<figcaption>Yes, we've all been there. Yes, it's really this much of a nightmare with dates and times.</figcaption>

</figure>

The Intl.DateTimeFormat object is a constructor for objects enabling language-sensitive date and time formatting.

At its most basic, the date time format takes in locales and options objects and outputs formatted time stamps based on those options.

Here's an example using locales from around the globe to format this date variable.

Intl.DateTimeFormat locales example

var date = new Date(Date.UTC(2012, 11, 20, 3, 0, 0));

// using US date time formatting
console.log(new Intl.DateTimeFormat('en-US').format(date));
// expected output: "12/19/2012" // using Great Britain date time formatting

console.log(new Intl.DateTimeFormat('en-GB').format(date));
// expected output: "19/12/2012"

// Include a fallback language, in this case US English with the Buddhist calendar
console.log(new Intl.DateTimeFormat(['en-US-u-ca-buddhist']).format(date));
// expected output: "12/19/2555"

In addition to being able to pass in locations, users can provide additional parameters strung together with the location, like:

nu — the number systems used in different countries,
ca — the calendar to use,
hc — hour cycle ( h11, h24, etc.).

After the additions to locales, comes the optional options object which can include properties like:

timezone — this can be either "UTC" or something like "America/New_York",
hourCycle — once again, specs like "h12" or "h23",
and my favorite one: formatMatcher — this algorithm can take in properties used to describe date-times and format the outputs to match.

Subsets for formatMatcher include things like:

weekday, year, month, day, hour, minute, second
hour, minute, second

And even then, the subset properties have different options. Here are the options for weekday.

"long" (e.g., Thursday),
"short" (e.g., Thu),
"narrow" (e.g., T).

Check out this example using DateTimeFormat's options arguments to format the outputs.

Intl.DateTimeFormat options arguments example

var date = new Date(Date.UTC(2012, 11, 20, 3, 0, 0));

// request a weekday along with a long date in Japanese
var options = { weekday: 'long', year: 'numeric', month: 'long', day: 'numeric' };
console.log(new Intl.DateTimeFormat('ja-JP', options).format(date));
// expected output: "2012年12月19日水曜日"

// an application may want to use UTC and make that visible in German
options.timeZone = 'UTC';
options.timeZoneName = 'short';
console.log(new Intl.DateTimeFormat('de-DE', options).format(date));
// expected output: "Donnerstag, 20. Dezember 2012, UTC"

// sometimes you want to be more precise in Australian English
options = {
  hour: 'numeric', minute: 'numeric', second: 'numeric', 
  timeZone: 'Australia/Sydney',
  timeZoneName: 'short'
};
console.log(new Intl.DateTimeFormat('en-AU', options).format(date));
// expected output: "2:00:00 pm AEDT"

// even the US needs 24-hour time for some occasions
options = {
  year: 'numeric', month: 'numeric', day: 'numeric',
  hour: 'numeric', minute: 'numeric', second: 'numeric',
  hour12: false,
  timeZone: 'America/Los_Angeles' 
};
console.log(new Intl.DateTimeFormat('en-US', options).format(date));
// expected output: "12/19/2012, 19:00:00"

// to specify options but use the browser's default locale, use 'default'
console.log(new Intl.DateTimeFormat('default', options).format(date));
// expected output: "12/19/2012, 19:00:00"

This is such a cool feature. It's going to make it so much easier to translate dates and times properly and format them in so many specific ways that packages like moment.js, made specifically for parsing, validating and manipulating JavaScript dates and times, will become unnecessary.

The next property I want to segue into, since I'm already on the subject of dates, is international relative time format.

International relative time format

The Intl.RelativeTimeFormat object is a constructor for objects that enables language-sensitive relative time formatting.

Similar to the Intl.DateTimeFormat, this object takes in a locales parameter and an options object.

The locales parameter functions exactly the same in this relative time format object as it does in the date time format, so I'll skip any further explanation here.

As for options, the Intl.RelativeTimeFormat has a variety of additional properties it accepts:

localeMatcher — a locale matching algorithm, with the options "best fit" or "lookup",
numeric — the format of the output message,
style — the length of the internationalized message.

style follows the same lines as weekday, in that it has three different values:

"long" (default, e.g., in 2 months),
"short" (e.g., in 2 mo.),
or "narrow" (e.g., in 2 mo.).

narrow and short could be the same depending on the country.

Below are some samples using Intl.RelativeTimeFormat in various locales and with various options passed to them.

Intl.RelativeTimeFormat examples

var rtf1 = new Intl.RelativeTimeFormat('en', { style: 'narrow' });
// relative time formats in English

console.log(rtf1.format(3, 'quarter'));
//expected output: "in 3 qtrs."

console.log(rtf1.format(-4, 'hour'));
//expected output: "4 hr. ago"

var rtf2 = new Intl.RelativeTimeFormat('de', { numeric: 'auto' });
// relative time formats in German

console.log(rtf2.format(2, 'day'));
//expected output: "übermorgen"

// Create a relative time formatter in your locale
// with default values explicitly passed in.
const rtf = new Intl.RelativeTimeFormat("en", {
    localeMatcher: "best fit", // other values: "lookup"
    numeric: "always", // other values: "auto"
    style: "long", // other values: "short" or "narrow"
});

// Format relative time in English using negative value (-1).
console.log(rtf.format(-1, "day"));
// > "1 day ago"

// Format relative time in English using positive value (1).
console.log(rtf.format(2, "day"));
// > "in 2 days"

Once again, super handy that we, as developers, don't have to deal with the logic of formatting days or hours or even some words, in multiple languages no less, according to the number passed in. No complicated if/else logic blocks or even ternary expressions.

With the help of Intl.RelativeTimeFormat just specify the language to use, the numeric output and the style and it's good to go. Nice! 😄

On that note of the Internationalization API, let's look next at another thing that varies quite a bit from language to language: how lists are formatted.

International list format

It's tough enough transforming an array in one language into something that resembles a human-readable list with ands or ors added in, try doing that in multiple languages.

The Intl.ListFormat object is a constructor for objects to enable language-sensitive list formatting.

As is becoming the recurring theme of this article, the first optional parameter ListFormat accepts is a locales specification.

The second object is options with parameters like:

localeMatcher — the locale matching algorithm, same options as with Intl.RelativeTimeFormat,
type — the format of the output message. Possible values are "conjunction", which stands for "and"-based lists (default, e.g., 1, 2, and 3), or "disjunction" that stands for "or"-based lists (e.g., 1, 2, or 3). "unit" stands for lists of values with units (e.g., 7 pounds, 11 ounces).
style — the length of the formatted message. Values include "long", "short", or "narrow".

Here's some examples of how you could use the Intl.ListFormat object in practice.

Intl.ListFormat examples

const vehicles = ['Cat', 'Dog', 'Fish'];

// list formats in English
const formatter = new Intl.ListFormat('en', { style: 'long', type: 'conjunction' });

console.log(formatter.format(vehicles));
// expected output: "Cat, Dog, and Fish"

// list formats in French
const formatter2 = new Intl.ListFormat('fr', { style: 'short', type: 'disjunction' });
console.log(formatter2.format(vehicles));
// expected output: "Cat, Dog ou Fish"

// list formats in Spanish
const formatter3 = new Intl.ListFormat('es', { style: 'narrow', type: 'unit' });
console.log(formatter3.format(vehicles));
// expected output: "Cat Dog Fish"

const list = ['BMW', 'Volvo', 'Audi'];

// list formats in British English
console.log(new Intl.ListFormat('en-GB', { style: 'long', type: 'conjunction' }).format(list));
// expected output: "BMW, Volvo and Audi"

console.log(new Intl.ListFormat('en-GB', { style: 'short', type: 'disjunction' }).format(list));
// expected output: "BMW, Volvo or Audi"

console.log(new Intl.ListFormat('en-GB', { style: 'narrow', type: 'unit' }).format(list));
// expected output: "BMW Volvo Audi"

How awesome is it that finally, finally there's a built-in API that can make lists look good? Heck, I'd take this even if it was only available for English-speaking countries or English language-only applications, this is so useful.

But wait! I've got another incredibly useful constructor to introduce: the number formatter! 🤑

International number format 💰

<figcaption>Make it rain, everybody.</figcaption>

</figure>

Although the Intl.NumberFormat definition doesn't sound exciting at first, the implications are big. This object is a constructor for objects that enables language sensitive number formatting. And you know what that means? The formatting for international currencies is a breeze.

In addition to locales and nu — the numbering system to be used, the Intl.NumberFormat object can accept the following options parameters:

localeMatcher,
style — the formatting style to use. Possible values are "decimal" for plain number formatting, "currency" for currency formatting, and "percent" for percent formatting; "decimal" is the default style.
currency — the currency to use in currency formatting,
currencyDisplay — how to display the currency formatted. The options are "symbol" to use a localized currency symbol such as €, "code" to use the ISO currency code, or to use a localized currency name such as "dollar".
useGrouping — Whether to use grouping separators, such as thousands separators or thousand/lakh/crore separators. Possible values are true and false.

There's a few more options around digits: minimumIntegerDigits, maximumSignificantDigits, but I think their usefulness is limited, so I'll skip defining them all here.

Intl.NumberFormat examples

var number = 123456.789;

// currency formatting with euros in German
console.log(new Intl.NumberFormat('de-DE', { style: 'currency', currency: 'EUR' }).format(number));
// expected output: "123.456,79 €"

// currency formatting in Japanese where the Japanese yen doesn't use a minor unit
console.log(new Intl.NumberFormat('ja-JP', { style: 'currency', currency: 'JPY' }).format(number));
// expected output: "￥123,457"

// limiting an Indian currency format to three significant digits
console.log(new Intl.NumberFormat('en-IN', { maximumSignificantDigits: 3 }).format(number));
// expected output: "1,23,000"

// Arabic in most Arabic speaking countries uses real Arabic digits
console.log(new Intl.NumberFormat('ar-EG').format(number));
// expected output: ١٢٣٤٥٦٫٧٨٩

// India uses thousands/lakh/crore separators
console.log(new Intl.NumberFormat('en-IN').format(number));
// expected output: 1,23,456.789

// the nu extension key requests a numbering system, e.g. Chinese decimal
console.log(new Intl.NumberFormat('zh-Hans-CN-u-nu-hanidec').format(number));
// expected output: 一二三,四五六.七八九

// when requesting a language that may not be supported, such as
// Balinese, include a fallback language, in this case Indonesian
console.log(new Intl.NumberFormat(['ban', 'id']).format(number));
// expected output: 123.456,789

In my mind, for an international company or website, this has got to be a huge, huge selling point for the Internationalization API. Instead of hand-coding for all the varieties of currency differences that exist, we can rely on a single API to handle it for us — what a logic and time-saver!

Ok, time for the last property I'll cover today: Intl.PluralRules.

International plural rules

When you think about rules for plurals in English, the rules are hard: "one dog" becomes "two dogs" but "one goose" becomes "two geese", "one" becomes "two" but beyond "two" becomes "a few" or "many" or "other" when describing some vague, larger numbers of items, and so on and so forth— are you starting to see where I'm going with this?

Now imagine needing to know unwritten rules like this in more than one language...ugh. 😑

Well, this is where the International Plural Rules becomes a life saver. The Intl.PluralRules object is a constructor for objects enabling plural sensitive formatting and plural language rules.

This constructor accepts locales (of course), and its options object takes in the following parameters:

localeMatcher — The locale matching algorithm to use. Possible values are "lookup" and "best fit"; the default is "best fit".
type — the type to use. Possible values are: "cardinal" for cardinal numbers (referring to the quantity of things). This is the default value. Or "ordinal" for ordinal number (referring to the ordering or ranking of things, e.g. "1st", "2nd", "3rd" in English).

Uses for Intl.PluralRules

There's a few different ways to use Intl.PluralRules in your code.

Basic usage — In basic use without specifying a locale, a formatted string in the default locale and with default options is returned. This is useful to distinguish between singular and plural forms such as "cat" and "cats".

Locales usage — When using locales, the example code below shows some of the variations in localized plural rules. In order to get the format of the language used in the user interface of your application, make sure to specify that language (and possibly some fallback languages) using the locales argument.

Options usage — The plural results can be customized using the options argument, which has one property called type which you can set to ordinal. This is useful to figure out the ordinal indicator, e.g. "1st", "2nd", "3rd", "4th", "42nd" and so forth.

The code snippets below outline all the usages I just described.

Intl.PluralRules examples

// basic usage examples
var pr = new Intl.PluralRules();
// examples of plural rules if in US English locale

console.log(pr.select(0));
// expected output: 'other'

console.log(pr.select(1)); 
// expected output: 'one'

console.log(pr.select(2));
// expected output: 'other'

// examples using `locale`
// for example: Arabic has different plural rules
console.log(new Intl.PluralRules('ar-EG').select(0));
// expected output: 'zero'

console.log(new Intl.PluralRules('ar-EG').select(1)); 
// expected output: 'one'

console.log(new Intl.PluralRules('ar-EG').select(2));
// expected output: 'two'

console.log(new Intl.PluralRules('ar-EG').select(6));
// expected output: 'few'

console.log(new Intl.PluralRules('ar-EG').select(18));
// expected output: 'many'

// examples using `options`
var pr = new Intl.PluralRules('en-US', { type: 'ordinal' });
// examples using US English and `options type` specification

console.log(pr.select(0));
// expected output: 'other'

console.log(pr.select(1));
// expected output: 'one'

console.log(pr.select(2));
// expected output: 'two'

console.log(pr.select(3));
// expected output: 'few'

console.log(pr.select(4));
// expected output: 'other'

Pretty nice, huh? I'll admit, of all the Intl methods, this is less widely applicable than some of the other methods included, but I'm sure it will still have its moments where it's just the ticket.

It is also worth noting that this method is still in "Draft" status, with the comment "initial definition" in MDN's docs, so I would hold off for a while longer before attempting to rely on it completely in production.

Conclusion

Translating websites and applications for use in different countries and languages is tough enough, but being able to correctly handle things like date time formats, currencies and lists of items with their many language-specific rules is a real challenge.

The newest version of the ECMAScript Internationalization API aims to make some of these arduous tasks easier by letting its methods do the formatting for you, based on locales and whatever other options you might want to specify.

For JavaScript developers working with international requirements, this stands to be a huge boon, and I for one, am looking forward to incorporating these new features into my own applications just for the total convenience these properties provide.

Check back in a few weeks, I’ll be writing about JavaScript, ES6 or something else related to web development.

Thanks for reading, I hope you see the value the ECMAScript Internationalization API can provide to your applications, and are able to incorporate some of the properties into your web apps in the future. Please share this with your friends if you found it helpful!

References & Further Resources

Viewport Units, the CSS You Didn't Know About But Should

Sun, 23 Jun 2019 00:00:00 GMT

import CodePen from '../../src/components/CodePen.astro';

<figcaption>Source: The whole Internet and every front end web developer at one time or other, let’s be real here.</figcaption>

</figure>

Introduction

CSS can be both the greatest thing in the world and the most frustrating thing in the world for a web developer at exactly the same time.

It is a joy when it works the way we envision it — it’s beautiful, it makes the site intuitive, accessible, possibly animated — just good in all the ways. But the effort, time and the pain of getting everything pixel perfect can be totally maddening to the point where we want to tear our hair out and the meme above is very, very real to us.

To be honest, I fall more towards the side of “I like CSS, and all that it does for me, even when it takes forever to get it there,” whereas some of my coworkers have less positive feelings about it. I don’t normally get to dive deep into things like animations, multi-browser support or some of the really bleeding-edge CSS stuff because the software I work on day-in and day-out is an internal application to help other employees at my company do their jobs better.

But I still really enjoy (usually) when I pick up a front end feature that gives me the opportunity to dust off my CSS skills and implement some transitions or hover states, or learn some new cool CSS tricks I’ve not used before.

Today, I want to share something I’ve been learning more about lately: CSS viewport units. They’re awesome for so many reasons.

CSS Viewport units

Viewport units have been around in CSS for a few years now, but I don’t see them used all that often, and I think part of it is because people just aren’t that aware that they exist. And there’s so much to learn and do in CSS, I’m not terribly surprised that’s the case. I’m sure there’s a ton of CSS tricks I’m not aware of either.

In case you haven’t heard of viewport units, let me give you a quick overview of what they are.

They are truly responsive length units in the sense that their value changes every time the browser resizes. — CSS Viewport Units: A Quick Start

If that doesn’t immediately make sense or sound incredibly useful to you right off the bat, bear with me, there’s some cool examples coming up.

But first, let’s define viewport units in terms of their CSS.

Viewport Height (vh)

This unit is based on the height of the viewport. A value of 20vh is equal to 20% of the viewport height.

Viewport Width (vw)

This unit is based on the width of the viewport. A value of 10vw is equal to 10% of the viewport width. If a browser window is 1200px wide, the viewport width will be 120px. Likewise, if a mobile view browser is 480px wide, the viewport width would be 48px.

These next two are a bit more interesting than the standard vh and vw, but also not as commonly used either.

Viewport Minimum (vmin)

The viewport minimum is calculated as a percentage of the viewport width or height, whichever is smaller at the time. 20vmin will resolve to 20% of the current viewport width in portrait orientations, and 20%of the viewport height on landscape orientations.

Viewport Maximum (vmax)

For viewport maximum (like viewport minimum), it is based on a percentage of the viewport width or height, whichever is larger at the time. 15vmax will resolve to 15% of the current viewport height in portrait orientations, and 15% of the viewport width on landscape orientations.

Personally, I don’t find the last two as useful as the first, but that’s just me.

Now, after making sure we’re all on the same page, let’s get to the interesting parts — the examples.

Benefits of using viewport units

At this point, you might be thinking to yourself, ok, that’s cool, but how does this differ or improve upon the percentages and media queries I use in my CSS today?

I’m so glad you asked...

Example 1: Child components can be larger than their parent components

If you look at the CodePen example above, you’ll see that traditionally, if a child div is using percentages to calculate its size, it bases those percentages off of its parent component.

Using viewport units, however, makes it possible to make child components larger than the parent component — they base their size off of the size of the viewport instead of their parent component.

Helpful? In certain situations, you betcha.

Example 2: Background images can be responsive without media queries

The example here shows the power of both viewport height and viewport width. If you look at the images and colors set as backgrounds, you’ll notice that when the browser resizes, the images (and text centered in them), resize right along with them because it’s using vh and vw to set the height, width and padding.

.fullscreen {
  width: 100vw; /* this will stretch the width of the viewport */
  height: 100vh; /* this will stretch the height of the viewport */
  color: white;
  padding: 40vmin 1rem;
}

No muss, no fuss, just perfectly centered, resizing backgrounds. Tell me that’s not awesome.

Example 3: Font Sizes can Scale with the Help of calc()

Another really cool thing that can be done with vw and vh is with calc(). By using calc(), font-sizes and headers can responsively resize based purely on the viewport.

In the example above, you’ll notice the h1 and the p text font-sizes automatically resize and grow accordingly with the help of calc().

h1 {
  padding: 10vmin 1rem;
  font-size: calc(32px + 1.5vw);
  background-color: #BF6FE5;
}
p {
  font-size: calc(16px + 0.5vw);
  margin: 0 0 10vmin 0;
}

And the padding for the h1 tag also uses the vmin tag in addition to the 1rem static padding, so regardless of whether the height or width of the browser is greater, it will always have some padding keeping it from running up against the edges of the viewport.

I know these may seem like little things, but just imagine the CSS developers are saved from writing based on improvements like this.

If you like these kinds of examples, I’d highly recommend the CSS articles at the bottom of this post to see more of the possibilities that can be achieved with CSS viewport units.

Conclusion

CSS gives websites the ability to be unique, beautiful, useful and delightful, and it can sometimes make the engineers who use it want to quit their jobs.

Ultimately though, it feels so rewarding to me when I get it right and users tell me how much better an application makes their jobs. CSS viewport units, though often overlooked in favor of flashier frameworks like flexbox and grid, have some super useful applications that can reduce the amount of code needed to make a site responsive and child elements independent of their parents in some ways

It definitely has a place in my CSS arsenal, now that I have a better understanding of its usefulness.

Check back in a few weeks, I’ll be writing about JavaScript, ES6 or something else related to web development.

Thanks for reading, I hope you can find some ways to incorporate CSS viewport units into your own sites, and (hopefully) reduce the amount of CSS needed along the way.

References & Further Resources

Github Templates: The Smarter Way to Formalize PRs Among Dev Teams

Tue, 11 Jun 2019 00:00:00 GMT

</figure>

Introduction

Today I’m going to give you a better, easier, more efficient way to keep pull requests not only uniform but also completely customized to your dev team.

Some common issues of a dev team

If you’ve ever worked with GitHub as a software developer, either solo or as part of a team, you should be familiar with the issues I’m about to show you.

Leaving less than stellar commit messages

First, the nonsensical commit messages. These are usually sent multiple times because we, as developers, are lazy and will keep pushing changes to GitHub with the same note even though the code update has nothing whatsoever to do with that message.

<figcaption>This is a typical bunch of commit messages, I found in one shared repo my team owns.</figcaption>

</figure>

NOTE:: If you want to see some laugh out loud commit messages, I highly recommend the website What the Commit. Just keep refreshing the page for a constant stream of hilarious (and accurate) commit messages.

Completely puzzling pull requests

You’ve seen (and/or submitted) a pull request like this one to a repo — don’t try to deny it. Names and discerning details have been omitted to protect the privacy of the guilty.

<figcaption>Excellent. This definitely tells me just what I need to know.</figcaption>

</figure>

Yes, this is an actual pull request on GitHub, titled only as “catching up” with a ton of commit messages that give very little context and no other details to anyone reviewing this any idea as to what was going on here.

At this point, you may be thinking to yourself that this does, in fact, resemble things you’ve seen (or submitted) before and you might not see a huge problem with it.

If you’re a solo developer working on your own projects, always aware of everything in the code base, I might be inclined to agree with you. However, if you’re on a team with 10 or 20 other developers and building an application used by hundreds or thousands of people, where other developers have to look at the changes you made, determine if the code and logic is sound and it won’t break anything in your main code branch, this is a big, big problem.

Temporary solutions - don't both using these options ⚠️

To solve these problems there are a few different solutions that my development team tried in the past:

Option 1: A completely manual process

A loose collection of requirements we had to manually add to every new pull request. Things like:

Links to the feature story we were working on in our backlog,
Links to the automated Jenkins branch where our feature branch was built and deployed into QA,
Links to the feature branch for testing,
A code coverage report,
Links to related PRs in other repos, etc.

This worked OK, except people commonly forgot (or just didn’t care to add) one or more of these links, and we didn’t have a formalized system for new devs joining the team to know how to format their PRs.

Option 2: A template that needed to be copy/pasted into every PR, every time

Our second attempt was slightly less manual: A pull request template that my team kept in a Slack .txt file pinned in one of our development channels. Here’s what that looked like:

## Story
Jira, Pivotal Tracker, (link to where the story you worked on lives)

## PR Branch
URL to the automated branch build for feature testing

## Code Coverage
URL to the automated code coverage report run during the automated build process

## e2e
URL to the automated end to end tests run against the feature branch

## UX Approved
yes / no

## Newman
yes / no

## Related PR's
TBD

But once again, devs had to go find the template, copy the markdown formatting, and paste it into each and every pull request opened up and fill out the contents of it.

This worked better because the template, at least, ensured there was less chance of forgetting something, but it still wasn’t foolproof. Oftentimes (because devs are people and people are lazy, myself included), we’d just go find an old closed PR, copy its contents and paste it into the new PR we were working on. That was OK until someone forgot to update the story link or the end to end test link... you see the issue.

But that was the best solution anyone had until our team had some fresh developers brought on board.

As I was showing one of our new devs the PR template we used (by copying it from out of another closed pull request), he asked why we didn’t just use GitHub Templates, and I said, “because none of us had ever heard of them.” Then, he showed me something really cool and very easy to do.

A better solution - what we're using now (and you should too) ⭐️

Option 3: GitHub Templates - the automated way to do PRs

GitHub templates are an awesome invention from GitHub for both pull requests and issues. Since PRs are what concerned my team, I found the GitHub docs to learn more.

Here's what I learned from the docs:

"When you add a pull request template to your repository, project contributors will automatically see the template’s contents in the pull request body." — GitHub Docs

This was just the sort of thing my team needed. It was a way to ensure every pull request was uniform without our devs having to think about it. Perfect.

Turns out, it was incredibly easy to set up as well.

Setting up GitHub Templates in a project

Although setting up a PR template is very straightforward, I’m happy to lay out the steps here for convenience sake.

Step 1: Add a .github/ folder into the root of your project

The docs say you can either add the PR template in the root directory itself or in a folder named docs/ or .github/.

Since we wanted to ignore the PR folder the majority of the time we were doing development, I chose to create the template inside of the hidden .github/ folder.

<figcaption>Here’s a shot of my repo with the hidden .github/ folder at the root of the project.</figcaption>

</figure>

Step 2: Add your pull_request_template.md markdown file inside the folder

In order for Github to automatically pick up the formatting from the markdown file for PR templates, you must name the file pull_request_template.md.

Below is an example of the one my team uses:

# <Feature Title>

Tracker ID: **_#ADD LINK TO PIVOTAL STORY_**

Unit tests completed?: (Y/N)

PR Branch
**_#ADD LINK TO PR BRANCH_**

Code Coverage & Build Info
**_#ADD LINK TO JENKINS CONSOLE_**

E2E Approved
**_#ADD LINK TO PASSING E2E TESTS_**

Windows Testing
**_#HAS WINDOWS BEEN TESTED?_**

Related PR
**_#ADD ANY RELATED PULL REQUESTS_**

UX Approved
**_#ADD UX APPROVAL IF NEEDED_**

Step 3: Add these files to your main branch and prepare for smooth sailing in all PRs going forward

Here’s a screenshot of a repo I have with the pull request template added to it, and an example pull request I raised.

<figcaption>Here’s my automated PR automatically generated when I open a new PR in this project.</figcaption>

</figure>

It really is that simple to have a standardized PR template included in any repo with very little effort.

Conclusion

When you’re working with a team of developers on a mission-critical product, pull requests are a necessary evil, but they can be less painful with the help of GitHub’s pull_request_templates.

The benefits of the formatted PR templates are many, but some of the highlights in my mind are how flexible they are; no matter what your team deems necessary to review and accept, a pull request can be easily included and implemented in any repo. That’s the kind of ease developers need for required formalities like PRs.

Check back in a few weeks, I’ll be writing about JavaScript, ES6 or something else related to web development.

Thanks for reading, I hope this helps you better manage your own team’s pull requests in an easy and more orderly fashion.

References & Further Resources

Use ES6 To Destructure Deeply Nested Objects in JavaScript

Mon, 27 May 2019 00:00:00 GMT

</figure>

Introduction

In case this is your first time reading one of my posts, thanks for taking the time, I am a full stack software developer with a fondness for JavaScript, and the React framework, in particular.

For just over a year, in my free time after work, I’ve been teaching myself React with the help of online documentation, articles, tutorials and building (and breaking) side projects.

With all these sources of knowledge and inspiration, I managed to build a user registration application that I was pretty proud of. It had a React front end, an Express/Node.js server back end, a MySQL database, it used Passport.js and JSON Web Tokens to handle authentication, the Sequelize ORM to perform CRUD operations on the database, a password reset feature through Nodemailer, Swagger endpoint testing on the REST API endpoints, a docker-compose.yml file to spin up all three pieces of the application simultaneously in a virtualized container environment, etc., etc. Have you had enough technology buzzwords thrown at you, yet? 😫

<figcaption>Yes, I threw out all those buzzwords, no, I‘m really not that big of a deal. 😉</figcaption>

</figure>

Please understand, this project was by no means achieved in a day, it started as a very basic idea and mushroomed in scope as I made forward progress and got more ambitious to see how far I could push it. But by the time I’d gotten to that point, I was feeling pretty proud of myself. I’d made a cool, useful little side project — completely built in JavaScript.

Then I introduced Airbnb’s ESLint configuration into the mix, and was humbled as the VS Code browser lit up left and right with red squiggles of disapproval. 😞 Instead of quitting though, I persevered in learning the best practices of modern JavaScript/ES6 development (according to ESLint and Airbnb), fixed the majority of my errors and became a better developer along the way.

There was one error, however, which eluded me. The deeply nested, destructured object. How, pray tell, do you destructure a prop object that’s several levels deep in JavaScript to ESLint’s satisfaction?

Today, I will show you how to use ES6 to destructure nested objects, my friends, AND what’s more, prevent the dreaded undefined error if an object is missing that property entirely.

Let’s get to it.

The issue: ES6 object destructuring & deeply nested objects

To give you some insight into what I’m describing, let me back up just a minute and go over what object destructuring in JavaScript is, and why it’s a little daunting once your objects get complex.

Object Destructuring in a Nutshell

As per usual, MDN’s documentation sums up JavaScript destructuring nicely:

The destructuring assignment syntax is a JavaScript expression that makes it possible to unpack values from arrays, or properties from objects, into distinct variables. — MDN Docs, Destructuring Assignment

This is a feature introduced in ES6, and what it means, in practice, is that your code can go from looking like this:

Plain old JavaScript object

const myObject = { 
  a : "foo", 
  b : false, 
  c : 11 
};

Traditional JS object property access example (dot notation and bracket notation)

With the object’s properties being accessed through dot notation or bracket notation like this:

console.log(myObject.a); // "foo"
console.log(myObject[b]); // false
console.log(myObject.c); // 11

ES6 object property access example (destructured)

To being accessed like this:

const { a, b, c } = myObject;
console.log(a); // "foo"
console.log(b); // false
console.log(c); // 11

That’s pretty cool, right? It’s shorter, it’s much more concise to read, and it makes pretty good sense logically.

There’s a lot more cool things that can be done with both arrays and object and destructuring, and I’d recommend you read the docs for more examples.

Ok, so now that ES6’s destructuring has been defined, let’s look at my issue, which seemed a lot less straightforward with how to approach it (at least, at first pass, to me).

A deeply nested object

Below is an example of the object I was working with in my React application.

this.props.match.params.username; 
// what a mouthful! no one wants to write this anymore

This was my nested object, written in traditional JavaScript dot notation syntax.

To give a little more context around it, this object was being taken from the URL string and passed into the HTTP call from the browser’s client to the server side to identify (and validate) the username in the database. This syntax is courtesy of the react-router-dom npm package and React’s props.

Regardless of where the object’s syntax and structure originated from, it’s complicated. It’s technically nested 4+ layers deep in this object to get to the useful piece of information, the username.

And unlike, const {a} = myObject;, trying to figure out how to dive several levels deep into an object was a little befuddling.

The ESLint errors: merciless as always

But did ESLint care?

<figcaption>I never imagined I’d have reason to use a honey badger meme in a blog post, and yet, here it is.</figcaption>

</figure>

In the immortal words of honey badger: “nope”.

As soon as ESLint saw this type of syntax in my code, sirens went off and the ESLint error showed up: Must use destructuring props assignment eslint(react/destructuring-assignment).

<figcaption>Thanks for being so helpful, ESLint, this is an extremely useful warning message. 🙄</figcaption>

</figure>

I visited the ESLint documentation, made a few half-hearted attempts to figure out how to pull this convoluted mess of objects apart, but eventually gave up and resorted to the old * eslint-disable react/destructuring-assignment */ import at the top of my files instead of fixing it properly. And I moved on with my life. 😬

Then, a while later, another developer pointed me towards an article that showed me the light, while we were trying to decide how best to guard our code against throwing errors if environment variables weren’t specified during local development.

After reading it, things made much more sense. Here is what I learned.

How to fix the ESLint "Destructuring Props" error

Apparently, I was trying to make the solve for object destructuring more complicated than it needed to be. Here’s a breakdown (or build up) to my object, with destructuring along the way.

One JavaScript object

const myObject = {
  props: "Hello world"
};

The destructured version becomes:

const { 
  props 
} = myObject; 

console.log(props); // prints: "Hello world"

Right, that seems logical, access the property’s value in the object just by wrapping that property in curly braces.

Next, up...

One JS object within another object

For an object inside another object, like:

const myObject = { 
  props: { 
    match : "Some value" 
  }
};

The destructured version becomes:

const { 
  props : { 
    match 
  },
} = myObject; 

console.log(match); // prints: "Some value"

All right, so to access property values two levels deep, first wrap the original property inside the top level object (in this case props) in curly braces, then, inside that object, wrap the inner object, match, in another set of curly braces.

Ok, following so far...

One JS object within two more objects

For a slightly more complex object, like:

const myObject = { 
  props: { 
    match: {
      params: "A new value"
    } 
  } 
};

The new destructured version becomes:

const { 
  props: { 
    match: { 
      params 
    }, 
  }, 
} = myObject; 

console.log(params); // prints: "A new value"

Once more, stepping through each tier: props gets wrapped in the first set of curly braces, match, the object contained within props is wrapped in the second set of curly braces, and finally, the third object params (with the value I want to access) is wrapped in the third set of curly braces.

The pattern is starting to make sense when deconstructed object by object.

One JS object within three. More. Objects

So, for an even more complex object, like this:

const myObject = { 
  props: { 
    match: { 
      params: { 
        username: "Paige" 
      } 
    } 
  }
};

The new destructured version becomes:

const { 
  props: {
    match: {
      params: { 
        username 
      }, 
    }, 
  }, 
} = myObject; 

console.log(username); // prints: "Paige"

The same rules and pattern as above still apply, just keep going one level further with the curly braces to reach the object properties you want to read.

To fix the ESLint error my React application was throwing: Must use destructuring props assignment eslint(react/destructuring-assignment), this is the solution I came up with.

const {
  match: {
    params: { username },
  },
} = this.props;

Since ESLint only wanted object destructuring from the props object level and onwards, I was able to make my object slightly smaller than a truly, entirely destructured object. Access the match object inside this.props, access the params object inside match, and finally, access the value attached to username for the info I was seeking.

Broken down into steps like that, everything started making a lot more sense.

Nice! But then I thought of an edge case, what if, for some reason, one of these properties was missing? Ah, there’s a way to safeguard against undefined errors too within destructured objects.

How to prevent undefined errors for missing properties: default values

I can’t tell you the number of times I’ve thrown errors in local development because a property on an object I was trying to access was undefined, meaning the property (and its value) didn’t exist on that object. Not as an empty string, nor as an empty array or even a 0. It just flat out didn’t exist.

TypeError: Cannot read property ‘params’ of undefined. It’s so annoying — especially when it causes React to throw a giant error in the browser. 😠

But there’s a way around this with object default values. Another nice feature of ES6, in the case that the value unpacked from the object is undefined, is that that variable can be assigned a default value.

That’s a whole other blog post, but what this means for me, in the case of a missing property on a destructured object, is that instead of throwing a TypeError, I could provide a fallback value in the form of a string or other value, and I’d receive that value, which I could specifically look for and then move on from.

Here’s an example with an object with just one property.

Plain old JavaScript object

const myObject = { 
  props: "I only have props, nothing else"
};

And the destructured version:

const { props } = myObject; 

console.log(props); // prints: "I only have props, nothing else"

If I tried to access an undefined property within a destructured object, normally:

ES6 destructured object with no default props

const { 
  props: {
    match 
  } 
} = myObject 

console.log(match); // throws error: TypeError: Cannot read property "match" of undefined

But, if I set a default value to fall back on if that property doesn’t exist, within that destructured object, like so:

ES6 destructured object with default props

const { 
  props: { match } = { match: "No match"}
} = myObject
console.log(match); // prints: "No match"

And with an empty object, an empty array, a default string or integer, or a host of other things besides the dreaded undefined error, I can set up error handling in the DOM to display some sort of message to the user so they know what’s going on and how to fix the situation —a much better experience than an obvious React error screen with a cryptic message that something’s gone wrong.

So, here’s a rewritten version of the destructured object my app needed with a username from the URL string. If the username’s not present in the URL, this object structure will default to the string of "no match", which I can then check for, and display a nicer error message to help the user get back to a working place in the application.

const {
  match: { params: { username } = { username: "no match" } },
} = this.props;

Default values are one, nifty way to guard against undefined errors in your destructured objects.

Conclusion

ES6 is a great improvement to the JavaScript syntax, and Airbnb’s ESLint configuration is an excellent linter to identify code where the latest and greatest in ES6 could be used but hasn’t been.

My biggest complaint is that sometimes it isn’t helpful enough in explaining just how to fix the errors it finds. Luckily, I found a resource to better explain how to destructure more deeply nested objects, and even how to set default values to prevent undefined errors from being thrown in my React applications.

Readability, clean, concise code, and error prevention with default values to fall back to, what more could one ask for? I’m a fan of object destructuring, especially now that I know how to handle all kinds of different scenarios with it.

Check back in a few weeks, I’ll be writing about JavaScript, ES6 or something else related to web development.

Thanks for reading, I hope this helps you better understand the power of ES6 object destructuring and how to use it in your own JavaScript applications. Please share this with your friends if you found it helpful!

References & Further Resources

MERN, User Registration Github repo
MDN Documentation object destructuring
react-router-dom, npm
MDN Documentation, object default values

JavaScript's Async / Await versus Promises: The Great Debate

Sun, 12 May 2019 00:00:00 GMT

</figure>

The spark that ignited the firestorm

One day, just a few weeks ago at work, I was minding my own business, writing some React code, submitting pull requests, and another developer on my team, in essence, lobbed a grenade at our codebase.

He asked a seemingly benign question about why we were continuing to use promises in JavaScript versus the newer ECMAScript 17 version, async / await. 🚨 It seemed innocent enough, at first glance.

And let me tell you, normally, I’m all for using the new hotness in JavaScript (especially when it’s built into the language and supported by most modern browsers — looking at you for continued non-compliance, Internet Explorer), but even though I’ve heard the about the greatness of async / await, I hadn’t really used it up to that point.

I didn’t really see any benefits of async / await that outweighed using promises — they both ended up accomplishing the same thing: handling asynchronous data calls in a performant, consistent manner.

<figcaption>All promises, all the time.</figcaption>

</figure>

I like promises, I’m comfortable with the syntax of .then(), with the error handling of .catch(), with all of it, and switching required rewriting a lot of asynchronous calls our application currently makes. Before I uprooted all the hard work the team had written in traditional promise form, I wanted to find some reasoning beyond “it’s ES7, and it’s newer” to make the jump to async / await.

And so, I set off to do some learning.

Today, I’ll compare the benefits (and personal preferences) of choosing promises or async / await for your asynchronous data needs in your JavaScript application.

A short history of JavaScript's asynchronous data handling (or lack thereof)

Now before I go into detail about promises and async / await, I want to backtrack to a darker time in JavaScript, when asynchronous data fetching was more of a problem and the lesser solution was known as callbacks.

AJAX & callbacks

AJAX, which stands for Asynchronous JavaScript And XML and callbacks were an OG way of handling asynchronous calls in JavaScript. What it boils down to, is when one function is meant to be executed after another function has finished executing — hence the name "call back".

Callbacks in a nutshell

The first function normally returns data the second function needs to perform some sort of operation on, and unlike multi threaded languages like Java, the single threaded JavaScript event loop will push that first function to the call stack and execute it, pop that function call off the call stack once it’s completed it’s request, and continue running, pushing other operations that were waiting in the queue to the call stack in the meantime (which keeps the UI from seeming as if it’s frozen to the user).

Then, when the response from the other server comes back (the asynchronous data) it’s added to the queue, the event loop will eventually push it to the call stack when it sees the stack is empty, and the call stack will execute that response as the callback function.

This article isn’t about callbacks though, so I won’t go into too much more detail, but here’s an example of a what a JavaScript event with a callback function looks like.

Traditional JavaScript callback example

// the original function to call
function orderFood(food, callback) {
  alert(`Ordering my ${food} at the counter.`);
  callback();
}
// the callback once the order's up
function alertFoodReady(){
  alert(`Order's ready for pickup`);
}
// calling the function and adding the callback as the second 
// parameter
orderFood('burger', alertFoodReady);

This is a pretty simple example, and to be fair, it’s only showing the "happy path" — the path when the callback works and the food’s prepared and ready to be picked up.

What’s not covered here is the error handling, nor is it an example of what can happen when you have many, many asynchronous dependencies nested inside one another. This is fondly known among JavaScript developers as “Callback Hell” or “The Pyramid of Doom”.

Callback Hell

Below is an example of what Callback Hell looks like. This is a nightmare, don’t try to deny it. This happens when multiple functions need data from other functions to do their jobs.

<figcaption>A perfect example of callback hell: a callback, inside a callback, inside another callback for eternity.</figcaption>

</figure>

If you’d like to error handle that or even try to add some new functionality in the middle of that mess, be my guest.

I, however, won’t have anything to do with it, so let’s agree that AJAX and callbacks were once a way to handle asynchronous data, but they are no longer the de facto way. There’s much better solutions that have come about that I’ll show you next. Let’s move on to promises.

Promises, promises

Mozilla does an excellent job of defining promises, so I’ll use their definition as the first introduction to what a promise is.

A Promise is an object representing the eventual completion or failure of an asynchronous operation…Essentially, a promise is a returned object to which you attach callbacks, instead of passing callbacks into a function. — Mozilla Docs, Using promises

Here’s another example of an asynchronous callback function called createAudioFileAsync(). It takes in three parameters: audioSettings, a successCallback, and a failureCallback.

Traditional JavaScript callback example

function successCallback(result) {
  console.log("Audio file ready at URL: " + result);
}

function failureCallback(error) {
  console.log("Error generating audio file: " + error);
}

createAudioFileAsync(audioSettings, successCallback, failureCallback);

That’s a lot of functions and code for just one asynchronous data call.

Here’s the shorthand for that same function when it’s transformed using promises.

JavaScript promise example with callbacks

createAudioFileAsync(audioSettings)
  .then(successCallback, failureCallback);

Does that look nicer to you? It looks nicer to me.

But wait, there’s more. Instead of the success and failure callbacks both inside the .then(), it can be changed to:

Modern day JavaScript promise example

createAudioFileAsync(audioSettings)
  .then(successCallback)
  .catch(failureCallback);

And even this can be modernized one more time, by swapping the original successCallback() and failureCallback() functions for ES6 arrow functions.

ES6 arrow function promise example

createAudioFileAsync(audioSettings)
  .then(result => console.log(`Audio file ready at URL: ${result}`))
  .catch(error => console.log(`Error generating audio file: ${error}`));

This may seem like a minor improvement right now, but once you start chaining promises together or waiting for multiple promises to resolve before moving forward, having one single .catch() block at the end to handle anything that goes wrong within, is pretty handy. Read on and I’ll show you.

Pros of promises over callbacks

In addition to a cleaner syntax, promises offer advantages over callbacks.

Callbacks added with .then() even after the success or failure of the asynchronous operation, will be called, as above.
Multiple callbacks may be added by calling .then() several times. Each callback is executed one after another, in the order in which they were inserted (this is the chaining I mentioned earlier).
It’s possible to chain events together after a failure, i.e. a .catch(), which is useful to accomplish new actions even after an action failed in the chain.
Promise.all() returns a single Promise that resolves when all of the promises passed as an iterable have resolved or when the iterable contains no promises. Callbacks can’t do that.
Promises solve a fundamental flaw with the callback pyramid of doom, by catching all errors, even thrown exceptions and programming errors. This is essential for functional composition of asynchronous operations.

To back up a moment, a common need is to execute two or more asynchronous operations back to back, where each subsequent operation starts when the previous operation succeeds, with the result from the previous step. This can be accomplished with a promise chain.

In the olden days, we entered Callback Hell when callbacks depended on each other for information. See below (note also, the multiple failureCallbacks that had to be added for each callback).

Traditional, nested callback chain example

doSomething(function(result) {
  doSomethingElse(result, function(newResult) {
    doThirdThing(newResult, function(finalResult) {
      console.log('Got the final result: ' + finalResult);
    }, failureCallback);
  }, failureCallback);
}, failureCallback);

With the introduction of the promise chain, that "pyramid of doom", became what you see below (see the improvement of just one failureCallback instance now at the very end).

New promise chain example

doSomething()
.then(function(result) {
  return doSomethingElse(result);
})
.then(function(newResult) {
  return doThirdThing(newResult);
})
.then(function(finalResult) {
  console.log('Got the final result: ' + finalResult);
})
.catch(failureCallback);

And with the advent of ES6 arrow functions, that code becomes even more compact in the next example.

ES6 arrow function promise chain example

doSomething()
.then(result => doSomethingElse(result))
.then(newResult => doThirdThing(newResult))
.then(finalResult => {
  console.log(`Got the final result: ${finalResult}`);
})
.catch(failureCallback);

Not bad, huh?

Note that with arrow functions, the return statement is unnecessary to pass on the result, instead the result is returned via implicit returns instead.

Likewise, sometimes you need two or more unconnected promises to all resolve before moving on, which is where Promise.all() becomes a godsend.

Promise.all() example

var promise1 = Promise.resolve(3);
var promise2 = 42;
var promise3 = new Promise(function(resolve, reject) {
  setTimeout(resolve, 100, 'foo');
});
Promise.all([promise1, promise2, promise3]).then(function(values) {
  console.log(values);
});
// expected output: Array [3, 42, "foo"]

Simply by passing the three promises as an array to Promise.all(), the promise waited until all three had resolved before moving on to the .then() part of the statement.

I’d like to see callbacks do that gracefully.

Promise FTW?

This is the kind of code my team had been writing in our React application. It was clean, compact, easy to read (in my opinion), I saw nothing wrong with it. Then I investigated async await.

ECMAScript 17's new hotness: async / await

<figcaption>The promise of async / await. See what I did there? 😏</figcaption>

</figure>

Once more, I turn to Mozilla for the most succinct definitions of async and await.

The async function declaration defines an asynchronous function, which returns an AsyncFunction object. An asynchronous function is a function which operates asynchronously via the event loop, using an implicit Promise to return its result. But the syntax and structure of your code using async functions is much more like using standard synchronous functions...

An async function can contain an await expression that pauses the execution of the async function and waits for the passed Promise's resolution, and then resumes the async function's execution and returns the resolved value. — Mozilla Docs, Async Function

While that may make sense to you, I usually benefit from seeing code to really get the gist of it. Here’s a couple of code examples so you can see the differences for yourself.

Promise-based example

Here’s an example of a promise-based fetch() HTTP call.

function logFetch(url) {
  return fetch(url)
    .then(response => response.text())
    .then(text => {
      console.log(text);
    }).catch(err => {
      console.error('fetch failed', err);
    });
}

Ok, this seems pretty straightforward so far.

Async / await example

Here’s the async / await version of that same call.

async function logFetch(url) {
  try {
    const response = await fetch(url);
    console.log(await response.text());
  }
  catch (err) {
    console.log('fetch failed', err);
  }
}

And this too, seems pretty easy to understand - but in a more concise manner.

Pros of async / await over promises

So what’s the big deal?

What this seems to boil down to in practice, is that async / await is really syntactic sugar for promises, because it still uses promises under the hood.

<figcaption>It’s all promises in the end!</figcaption>

</figure>

The change to the syntax, though, is where its appeal to many starts to become apparent.

The syntax and structure of your code using async functions is much more like using standard synchronous functions.
In the examples above, the logFetch() functions are the same number of lines, but all the callbacks are gone. This makes it easier to read, especially for those less familiar with promises.
Another interesting tidbit, is that anything you await is passed through Promise.resolve() (for us, typically the .then(result) resolution of the promise), so you can safely await non-native promises. That’s pretty cool.
And you can safely combine async / await with Promise.all() to wait for multiple asynchronous calls to return before moving ahead.

To show another example of async / await which is more complex (and better demonstrates the readability of the new syntax), here’s a streaming bit of code that returns a final result size.

Second promise-based example

function getResponseSize(url) {
  return fetch(url).then(response => {
    const reader = response.body.getReader();
    let total = 0;

    return reader.read().then(function processResult(result) {
      if (result.done) return total;

      const value = result.value;
      total += value.length;
      console.log("Promise-Based Example ", value);

      return reader.read().then(processResult);
    })
  });
}

<figcaption>Me, at the first glance of the code above.</figcaption>

</figure>

It looks fairly elegant, but you have to stare at the code for a good bit, before you finally understand what it’s doing. Here’s that same code again but with async / await.

Second async / await example

async function getResponseSize(url) {
  const response = await fetch(url);
  const reader = response.body.getReader();
  let result = await reader.read();
  let total = 0;

  while (!result.done) {
    const value = result.value;
    total += value.length;
    console.log("Data received ", value);
    // get the next result
    result = await reader.read();
  }

  return total;
}

All right, now I’m getting a better idea of async / await syntax being more readable. That code is much easier read, I can agree with that. And the fact that I can combine it with Promise.all() is pretty nifty as well.

But to change all the promise-based calls in the entire codebase, I needed more convincing beyond readability for developers...I needed cold, hard, performance-driven benefits.

<figcaption>Convince me to refactor everything.</figcaption>

</figure>

The silver bullet moment: when async / await won the day

And here’s the article that really changed my mind, from the team that actually builds and maintains the JavaScript V8 engine running all my Chrome browsers right now.

The article sums up how some minor changes to the ECMAScript specifications and the removal of two microticks of time, actually lets “async / await outperform hand-written promise code now,” across all JavaScript engines.

Yes, you read that right. The V8 team made improvements that make async / await functions run faster than traditional promises in the JavaScript engine.

That was all the proof I needed. It actually runs faster in the browser? Well sign me up.

Conclusion

Promises and async / await accomplish the same thing. They make retrieving and handling asynchronous data easier. They eliminate the need for callbacks, they simplify error handling, they cut down on extraneous code, they make waiting for multiple concurrent calls to return easy, and they make adding additional code in between calls a snap.

I was on the fence about whether rewriting all our traditional promises to use async / await was worth it, until I read (from the maintainers of the JavaScript V8 runtime engine) that they’d actually updated the engine to better handle async / await calls than promises.

If there’s performance improvements for our application that can be gained from something as simple as a syntax change to our code, I’ll take a win like that any day. My ultimate goal is always a better end user experience.

Check back in a few weeks, I’ll be writing about JavaScript, ES6 or something else related to web development.

Thanks for reading, I hope this helps you make a better informed decision about which syntax style you prefer more: promises or async / await so you can go forward with a consistent asynchronous data handling strategy in your code base.

References & Further Resources

JavaScript Promises, MDN Docs
Using Promises, MDN Docs
Promise.all(), MDN Docs
Async Await, MDN Docs
AJAX, W3Schools Docs
JavaScript Event Loop, MDN Docs
Callbacks, JavaScript Info
V8 Developers Blog, Faster Async Functions and Promises

4 Solutions to Run Multiple Node.js or npm Commands Simultaneously

Thu, 02 May 2019 00:00:00 GMT

<figcaption>Node and npm: they just go together like peanut butter and jelly.</figcaption>

</figure>

Funny story: npm doesn't provide an immediately obvious way to run multiple scripts at once

We’ve all been there — you open up an app for local development and you need to run your Express/Node.js backend server and your React UI server simultaneously, but in order to do so you have to open up two terminal windows (or more), cd into two different package.json files via the command line and run two different start commands.

Sound familiar? If only there were a better way...

Quite simply, there is a better way. And like all good things in web development today, there’s a thousand different ways to achieve the same end.

Today, I will share four ways to run multiple Node.js commands or npm scripts at once.

Ready? Ok, let’s run through them.

The solution(s)

Option 1: Bash && chained commands

<figcaption>Good old Bash - can't beat the command line.</figcaption>

</figure>

This solution is the most straightforward and requires no extra npm packages or other software — it is literally just the command line shell.

Fun fact: the "scripts” in a package.json file are actually just terminal commands that would be run in an OS’s shell (like Bash). So by using standard shell syntax, you can chain together commands that NPM runs by calling the key associated with those combined values — see my example below.

Inside your npm script, in the "start" command, or whatever you want to use to run your servers (provided they’re in the same repo, of course), just chain together your two start scripts like so.

"scripts": {
  "start": "react-scripts start",
  "dev": "(cd server && npm run start) & npm run start"
}

The way this works, is that at the root level of the project is my React project (the "react-scripts start" command executes in the shell when you type in npm run start in the terminal), and one level inside of that is a folder called /server which holds the Node.js server used to proxy calls to all the backend microservices, and starts up with the same command at its root level thanks to its own package.json.

Here’s a very dumbed down diagram of the project structure, for reference:

root/ 
├── package.json 
├── server/ 
| ├── package.json

As you can see, both levels of the project have separate package.json files with their own dependencies and individual npm start scripts, but by simply chaining the two together with the & in between them, both Node commands can be run.

Additionally, using the && means that the script will wait until the server has successfully started before spinning up the client-side React application (because && in bash means the things on both sides of the && must evaluate to true). If the server can’t start for some reason, the whole npm command will fail and the UI won’t start without its backend server ready to go too.

That’s the quickest, easiest, most built-in way to run multiple Node.js commands in different file locations, at once.

Now, I’ll move on to a few npm packages that make the process even simpler, and require no knowledge of Bash.

Option 2: Concurrently

<figcaption>npm package #1: Concurrently</figcaption>

</figure>

The first npm package I’ll introduce you to is called Concurrently. Its name is pretty self-explanatory: it runs multiple commands concurrently. 😝

It’s not as robust, in terms of customization, as my next npm package recommendation, npm-run-all, but I think the majority of the time (hopefully) all the extra configuration is unnecessary.

After running npm i concurrently to install it, you can then set up your NPM start script to run multiple commands just by separating each individual command with quotes.

In a package.json file, your "scripts” command might look something like this, (note the need for escape quotes here):

"scripts": {
  "start": "react-scripts start",
  "dev": "concurrently \"cd server && npm run start\" \"npm run start\""
}

And once again, you should be off to the races. Also worth noting is you can run these same types of commands with quotes surrounding each argument from the command line as well, after Concurrently’s been installed globally.

I also recommend checking out the documentation for more cool tricks you can do with it, like shortening commands, supporting wildcards, etc.

Option 3: npm-run-all

<figcaption>npm package #2: npm-run-all</figcaption>

</figure>

Moving on, this package is another popular option from NPM, called npm-run-all.

The npm page proclaims npm-run-all to be "A CLI tool to run multiple npm-scripts in parallel or sequential."

It is a similar concept to how Concurrently works, but the syntax is slightly different and npm-run-all touts how it can shorten a very long, single start command like: npm run clean && npm run build:css && npm run build:js && npm run build:html

Into: npm-run-all clean build:*

The npm-run-all CLI is installed can be installed via npm or Yarn: npm install npm-run-all —-save-dev, and once installed, it boasts three different commands, based on your needs:

npm-run-all - the main command, which has documentation on all of the flags and additions you can pass in via the command line.
run-s - run sequential — for when you need one command to finish before the next one starts.
run-p - run parallel — like when both the UI and server pieces of the application need to run side by side.

If, for example, the package.json scripts looked like the following:

{
  "scripts": {
    "clean": "rimraf dist",
    "lint":  "eslint src",
    "build": "babel src -o lib"
  }
}

That could become with npm-run-all: npm-run-all clean lint build.

The combinations can also get much fancier with combinations of parallel and sequential runs together, depending on what your needs are. See the documentation for more details.

Pretty easy, right?

And now, on to the last option for running multiple npm commands at once, Docker.

Option 4: docker-compose

<figcaption>The final solution — and a little out in left field: Docker.</figcaption>

</figure>

Docker and Docker-Compose are whole other articles, which I’ve covered in detail, here and here on Medium. I’d recommend checking both of those out if you’re unfamiliar with the virtual containerization platform that is Docker. It’s an extremely powerful and effective tool, when used properly.

For this article, I’ll keep the Docker conversation focused specifically on the Dockerfile and NOT the docker-compose.yml, which is another piece of the Docker puzzle.

The Dockerfile provides all the instructions and commands a user could call on the command line to assemble a Docker image. Essentially, it defines an app’s environment so it can be reproduced anywhere.

The docker-compose.yml defines the services that make up the app, so they can be run together in an isolated environment. That’s a separate piece unrelated to the focus of this article.

NOTE: If you’d like a more in-depth explanation of using docker-compose to improve your application development in lower life cycles and even production, please see this article I wrote on the subject.

You can also see my MERN project repo, which uses docker-compose to spin itself up, complete with a MySQL database instance running locally.

In terms of the Dockerfile, though, after you’ve installed Docker on your machine, it’s very simple to write for a completely JavaScript project.

Here’s my file structure for the React UI (client/ folder) and Node.js backend (api/ folder) for reference of the file structure.

root/
├── server/ 
├── client/ 
├── docker/ 
├── docker-compose.yml 
├── Dockerfile

And here’s the contents of the whole Dockerfile.

Dockerfile

FROM node:9
WORKDIR /app
CMD ls -ltr && npm install && npm start

Here’s what happening in the lines above:

FROM node:9 - Download a version of Node.js from Docker hub.
WORKDIR /app - Define the working directory for each app (both go at the root of their respective containers as /app).
CMD ls -ltr && npm install && npm start - npm installs all the dependencies in each using their own package.json files, and once the dependencies are downloaded, start the apps (both have an npm start command in their "scripts").

That’s it. That’s all that’s needed because the docker-compose tells the multiple applications (server/ and client/) how to structure themselves, where to mount needed volumes, open ports to the outside environment, get the needed Docker images for things like databases, etc.

I’d recommend checking out my Docker-powered project to see more examples of what I’ve described above.

It’s a little bit of a different tack to run multiple npm scripts at once from the other solutions, with what is arguable a bit more initial overhead, but it works, and if you’re familiar with Docker’s many benefits, it’s a pretty sweet way to run your projects.

Conclusion

Whatever you choose as a resolution, this is a problem that every JavaScript developer runs into at some point or other.

You need a config file to run before your server starts, you need two servers to run side-by-side to handle UI views and API calls, you need watcher files to keep running while you make file changes — there’s a million reasons. Regardless of the exact scenario, eventually there will come a time when it will be beneficial to run multiple Node.js commands or npm scripts at the same time.

You could do this manually with multiple, open terminal instances, or you could use one of the solutions I suggest above. One is pure shell scripting, two are popular, stable npm packages, and one is the pretty far removed, but extremely powerful solution of using Docker to containerize and run multiple apps.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading, I hope this gives you some new ideas on how to approach running your own simultaneous npm scripts when the need arises.

References & Resources

Take Your VS Code Config Anywhere Easily with Settings Sync

Sat, 13 Apr 2019 00:00:00 GMT

<figcaption>VS Code — the best JavaScript IDE available today</figcaption>

</figure>

My optimum work setup, and how I play with fire

I’ve had my current, work-issued MacBook Pro for just under two years, and as you might imagine as a software developer, I’ve installed a lot of stuff on it. Some things, like tools I use regularly, I remember, but most things I’ve installed on it, I probably don’t. It’s the install-once-globally-and-then-never-have-to-think-about-it-again mentality.

And I know this is a dangerous game to play, because while I’m usually careful with my personal effects, things happen. Coffees get spilled, laptops slip to the floor, OSs get corrupted, and other people can be less conscientious with themselves and their stuff.

At that point, I have a problem, I have a new or freshly reset laptop devoid of all my personal settings, plugins, random installs and preconfigured tooling that makes development easier and more fun. Especially when it comes to my IDE (Visual Studio Code) — I have it tweaked and set up just how I like it, and in a blink, it could all be gone.

In that moment, I am reminded of just how painful set up and configuration is. Apparently, I must block this out of my memory every time I have to go through it, because let’s be honest, it’s just a big pain.

This article won’t walk through perfectly restoring your entire machine to the state it was before the incident, that’s beyond the scope of this post.

This article will, however, show you how to perfectly recreate your Visual Studio Code IDE settings without starting over from scratch and spending hours on it.

Let’s get going.

Settings Sync to the rescue

If you haven’t read any of my previous posts here and here about VS Code, I would highly encourage you to do so. It’s an incredible IDE that’s free AND it trumps WebStorm in almost every way, which is not free, I might add (and which I used before).

One of my favorite things about VS Code is the extension marketplace that’s absolutely chock-full of useful plugins people have rolled themselves and felt the need to share with the rest of us developers, mostly for free as well. Bless these people.

These extensions are some of the things that make VS Code such a delight to develop with — they help make it my own editor too. My color theme (Night Owl), my list of must-have plugins (Prettier with auto format on save enabled, among others), my sidebar of useful doohickeys (Docker, GitLens), not to mention Live Share and the ever-growing list of features the VS Code team keeps releasing each month. I could go on…

When faced with the possibility of losing (or even trying to transfer) my carefully developed VS Code setup to another machine, I knew there had to be a way to do it gracefully. Some enterprising developer somewhere had faced the challenge I was now facing and had figured out a way to sync their settings across machines.

I just knew the solution had to be out there, and so, I asked the internets, and it brought back Settings Sync (formerly known as Visual Studio Code Settings Sync).

Settings Sync

<figcaption>I wish this cute little cloud came in a larger image size for this post, but alas. </figcaption>

</figure>

🌴💪 Synchronize your Visual Studio Code Settings Across Multiple Machines using Github GIST 💪🌴 — Settings Sync, GitHub

This plugin, free in the VS Code marketplace is exactly what its Github description says it is: a tool that can sync your VS Code settings across as many machines as you please (complete with palm tree emojis and all), and Github’s gists make it possible.

And it syncs everything:

Settings file
Keybinding file
Launch file
Snippets folder
VS Code extensions & extensions configurations
Workspaces folder

Sounds good in theory, but is it really as easy as its made out to be? Actually, yes. because the documentation that accompanies the Settings Sync plugin is nothing short of great.

Ok, let’s get set up with Settings Sync.

Settings Sync in action

Like I said, the actual directions supplied by Settings Sync’s creator are pretty good, but since you’re here, I’ll go through them step-by-step and clarify a few points that I would have appreciated knowing the first time around.

Step 1: Install Settings Sync in VS Code

<figcaption>This is the official plugin you’ll see in your own VS Code marketplace.</figcaption>

</figure>

The obvious first step is to install Settings Sync into your VS Code terminal from the marketplace. This is the icon you’ll see when you search for it.

Next up, you’ll need to head to your Github account.

Step 2: Generate a Personal Access Token (PAT) from GitHub

The way that Settings Sync works is through Github, it makes a private gist with all the VS Code info saved there, which is then accessible to anyone else who has the keys to access the gist.

So in Github, you’ll go to: Settings > Developer settings > Personal access tokens > Generate New Token (see screenshot below).

<figcaption>Here’s a picture of my own VS Code settings sync gist.</figcaption>

</figure>

As you can see, I already have a “vscode-settings-sync” token, but for the purposes of this article, I clicked the “Generate new token” button to walk through the steps.

Once you’re in the generation of the token, name the token something easy to remember and click the checkbox next to “Create gists”. That’s all you have to do before you create the token.

<figcaption>All you have to click is the "gist" box when creating the token.</figcaption>

</figure>

After your new token is generated, copy the token hash to your computer clipboard (or Sublime or notes or somewhere), you’ll never be able to access this string again in the future.

<figcaption>Copy the access token hash for future use.</figcaption>

</figure>

With that done, we’re ready to head back to VS Code now.

Step 3: Upload your VS Code settings

<figcaption>Sync your settings with update / upload settings</figcaption>

</figure>

Once back in VS Code, open up the command palette with command + shift + p (for Macs), and start typing in sync: and you should see the following list of options. Click the first one: **Sync: Update/Upload Settings** and you’ll be prompted to add the GitHub token you just copied from your newly created gist.

<figcaption>This is when you enter the hash token from GitHub.</figcaption>

</figure>

Now you’ve entered the token, all your current VS Code settings should be uploaded, and the OUTPUT terminal in VS Code will show a message similar to the screenshot below.

<figcaption>Verify it worked from the output in the VS Code terminal. </figcaption>

</figure>

You can see my settings and extensions files were uploaded along with all of the extensions I’m currently using in my own VS Code setup.

You should also be able to go to your gists in Github and verify the settings are there. It will be stored in Gists under a file name like cloudSettings.

But before you close this output terminal: copy the GitHub token and Gist ID generated by the upload — you’ll need these to download your settings on other machines. Put it somewhere you’ll be able to access it from the new machine you’ll be downloading the settings to (Slack, Google Docs, etc.).

<figcaption>You need to keep this gist and token somewhere where you’ll be able to access it again on another machine, like Slack, Sublime, etc.</figcaption>

</figure>

Now, we’re ready to move on to the new VS Code IDE that needs these settings.

Step 4: Download your settings on a new machine

Start up your new VS Code machine, and you’ll follow the same first step as uploading the settings: open up the command palette with command + shift + p, then start typing in sync:, but this time, you’ll select the **Sync: Download Settings** option from the list.

<figcaption>Download the settings to your new VS Code set up this way.</figcaption>

</figure>

After you’ve selected that, Sync Settings will prompt you to enter first your Github personal access token — this is the token you copied from the terminal output created when you uploaded your VS Code settings to Settings Sync.

<figcaption>First enter the GitHub token.</figcaption>

</figure>

Then, you’ll be asked to enter the Gist ID, also provided from the output terminal (this is the part that tripped me up the first time, I didn’t realize I needed both the token and ID, the first time around). It resulted in my having to reset my Settings Sync and do it again from the original computer to generate a new token and ID, not a big deal, but a bit of a pain.

<figcaption>Then enter the gist ID.</figcaption>

</figure>

After that, your original settings should download to VS Code without an issue.

Step 5: Restart your editor and enjoy

You’re done! At this point, you may need to close the VS Code editor completely and reopen it for all the changes to take effect, but that should be it.

With one big thing knocked off your to-do list, getting your IDE right how you want it to be, you can get on with everything else that comes with setting up a brand new machine. If only it were all so easy. 😬

And here’s a final fun fact, if you want to share these VS Code settings across a team of developers and machines, you can even make a public Gist that they can all access, more info on that is available here, under the “Create Public Gist to Share Settings” section.

Conclusion

There’s nothing that feels as good when developing software (especially JavaScript) as a fully configured IDE like VS Code. And nothing more painful than trying to remember all the little nuances, plugins and tweaks you did to get it to that state of perfection.

Now, with the help of the Settings Sync plugin, you don’t have to remember all those things, or spend hours recreating them on a new machine, you can simply save them to Github cloud (and let’s face it, if Github goes down, we’ve all got bigger problems than what our VS Code configurations were) and access them from any other machine as long as you have the personal access token and Gist ID.

Check back in a few weeks, I’ll be writing about React or something else related to web development, so please follow me so you don’t miss out.

Thanks for reading, I hope this helps you safeguard yourself and your VS Code settings from future accidents, it only takes a few minutes to do, but it can save you hours of reconfiguration.

References & Further Resources

Visual Studio Code
Settings Sync documentation
Settings Sync extension in VS Code marketplace
Settings Sync, GitHub
VS Code Marketplace

Node Engines: Helping Developers Everywhere Avoid Phantom Bugs

Sun, 31 Mar 2019 00:00:00 GMT

</figure>

The classic opening line: “It works on my machine.”

Let me set the stage for you: you’re writing code, minding your own business, knocking out a shiny new feature for your web application when suddenly a red-light starts flashing (or the #support Slack channel lights up, either one) and it’s all hands on deck: there’s an issue in production. 🚨

As the development team drops everything to figure out what’s going wrong and attempts to pinpoint and recreate the issue, after twenty minutes of pointless debugging, one bold soul shrugs and says, "Works on my machine." 😒

<figcaption>Every developer has heard (and said) this at least eleventy billion times.</figcaption>

</figure>

This statement helps exactly no one, but accurately sums up the issues I’m about to discuss:

Something as simple as an incompatible Node version in production can break your application, and be nearly impossible for developers to locally figure out that’s the problem. Locking down your Node engine can help you quickly debug and avoid this.

You never give a second thought as to which version of Node or NPM your machine is using locally versus the version of Node.js your production environment is using… until it matters.

Real World Examples of Why This Matters

I speak from personal experience when I say, an AngularJS 1.5 application my team supports ONLY works in Node version 9, however, when we deployed to our production cloud environment, the Node buildpack that deployed with it, pulled in v6. Version 6?!? Who even develops with a Node version that old any more??

Regardless, our deployment failed because Node 6 couldn’t download the node_modules dependencies correctly, the application failed to start up, and it took several hours of pouring over code, build logs, and environment variables to figure out what the actual problem was. That deployment became known as "Dark Thursday" within my dev team. To this day, if you ask someone who was there about that night, they’ll give you a haunted look in return. 😱

Similarly, our React application needs Node version 10 or higher, to take advantage of all the ES6 and beyond node dependencies that it has. It needs to be at least v10 or above, no ifs, ands, or buts.

So now you can see why not knowing what version of Node your cloud buildpack will default to is bad. It means you don’t know if your node modules will successfully be downloaded or fail. We can all agree that’s no way to develop.

But how do we ensure that we get the buildpack we need? How do we guarantee when we’re deploying from our local development environment to our cloud production environment that the Node.js runtime is the same? That it can support the Node dependencies our project needs to run?

It’s actually easier than you might think: strictly defining the Node engines for a project. Read on to learn how.

How do we prevent the wrong version of Node taking down our application?

Easy: engines

<figcaption>I don’t blame you if this is your initial response, it was mine too.</figcaption>

</figure>

Engines? What does that even mean?

What are Node engines?

Node engines are a little discussed (but in my opinion pretty critical) configuration that can be specified in your package.json file that tells anyone (or any machine) running the JavaScript application which version of Node is required for the code to work.

npm’s own docs say when defining engines:

“You can specify the version of node that your stuff works on” — npm

Pretty self-explanatory.

What this means in practice is: if engines is included, when a JavaScript build deploys (depending on how the engine field is specified) it will look for a version of Node at or above the version configured in the package.json, then download it and install all the node_modules dependencies using that engine.

If an engine is not specified, the project is at the mercy of the buildpack gods, and will assume any old version of Node will do and download the dependencies with some random version of Node and npm. Which is how we end up with fire drills like the scenario described above.

In fact, you can actually specify the version of Node.js, the version of npm and the version of Yarn a project uses in its engines config, which is pretty sweet.

Just to be crystal clear: the engines field will be verified when you install the package, not when you run the application.

Now that you’re better acquainted with Node engines, why they’re useful and how they work, let’s go about setting up a project to prevent an easily avoidable deployment failure.

Step 1. Figure out your local development runtime

The first step is to figure out which version of Node you’re with developing locally. Simple enough.

Just open up a terminal window and type:

node -v

Then, you should see the version of Node you’re working with printed out to the terminal. See the screenshot below.

<figcaption>How to see the local version of Node running on your machine. For me, it’s currently v11.10.0.</figcaption>

</figure>

If you ever need to switch Node versions for local development, I’d highly recommend NVM (Node Version Manager), which I also wrote a blog post about here.

But that’s a tangent. All you need for now, is to know what Node version you’re currently developing in. Let’s move on to Step 2.

Step 2: Specify your Node engine in the package.json

Now that you know the version of Node, you can specify it in your package.json file in your project. Below are a couple of examples what the engines config looks like.

Node and npm version example

This example includes both a Node version that’s greater than or equal to v11.10.2 and less than v12.0.0, and an npm version of exactly 5.8.0.

... // code above here like dependencies and other specs
  "engines": {    
    "node": ">=11.10.2 <12.0.0",    
    "npm": "5.8.0"  
  },
... // more code below here

Node version example

Or this example, which includes a Node version around 10.15.0.

... // more code 
  "engines": {    
    "node": "~10.15.0"  
  },
... // more code

Specifying Node, npm and Yarn versions is similar to specifying them for dependencies in your package.json, you can be as specific as you want down to the exact version, provide a lower limit, provide a range or an approximate. It’s up to you.

After you’ve set the Node engine in the package.json, it’s time to deploy the application and verify the correct engine is being downloaded and used.

Step 3: Run your deployment build scripts & verify the buildpack matches

Once your build pipeline is deploying the JavaScript application, you should be able to verify in the logs that the specified Node version is being downloaded and the correct node module dependencies along with it.

Below are some logs from our Jenkins server that runs all of our builds for our AngularJS application, which requires a version of Node 9. As you can see in the first screenshot, the build is checking for a specific Node version on the server and finds 9.11.2 specified in the package.json, so it installs that version to the build server if it’s not already available.

<figcaption>Here’s verification that the version of Node.js is running on the Jenkins build server.</figcaption>

</figure>

And here’s another screenshot example, this time of our React application, which needs a Node version of 10 or above, downloading Node v10.15.0, also specified in the React’s package.json "engines" field.

<figcaption>Another example of the specified Node engine being downloaded to then get the project’s needed node module dependencies.</figcaption>

</figure>

After this happens, the specified Node dependencies can be successfully downloaded and our applications should be able to start up without a problem.

Step 4: Enjoy your less stressful production deployment

Feel the relief wash over you as you suddenly have one less variable to worry about when you deploy to production. 😄

</figure>

Conclusion

Node engines won’t solve all your problems, but they at least should take some of the guesswork (and unintended bugs) out of your production deployments. For the small amount of work it takes to specify this in your package.json configuration, the investment now can save you hours down the line. Trust me, it’s worth it.

Check back in a few weeks, I’ll be writing about JavaScript, React or something else related to web development.

Thanks for reading, I hope this convinces you to specify your Node versions and npm / Yarn engines in your package.json configuration.

References & Further Resources

npm, Engines documentation
NVM, GitHub

Update Feature Toggles in a React App without Redeploying

Fri, 15 Mar 2019 00:00:00 GMT

</figure>

Feature toggles: the preferred way to test new UI features

Have you ever said to yourself: “Gee, I wish I could deploy this new feature to just a few of my users at a time. I want to test if they like it and it works well, before I open it up to the entire user base.”?

Then you shrug, deploy to production, and the site crashes when 250 users find that one unhappy path you didn’t even contemplate it was so far outside the scope of how to logically use the site (because that’s what users excel at).

This kind of scenario (and the ensuing mad scramble to code up a fast fix) could have been prevented with feature toggles.

So what are feature toggles?

Aside from the scenario I described above, according to Wikipedia:

A feature toggle is a technique in software development that attempts to provide an alternative to maintaining multiple source-code branches (known as feature branches), such that a feature can be tested even before it is completed and ready for release. Feature toggle is used to hide, enable or disable the feature during run time. For example, during the development process, a developer can enable the feature for testing and disable it for other users.

— Wikipedia, feature toggle

Feature toggles allow developers to introduce new features to a small group of users (beta testers), ensure that all the kinks and bugs are worked out, and the feature is solid and well liked by the users before letting everyone have access to it.

It also allows for multiple versions of an application to exist at once. Although feature toggles are generally thought of as impermanent - used for testing features then releasing them to the general public once they’ve been vetted - they could also be used maintain multiple versions of an application, with different versions available to different users.

If you’d like to read more about how to go about setting up a config server, you can read an article I wrote a while back about setting one up with the help of Spring Boot’s Cloud Configuration package.

And if you’d like to see another way of setting up feature toggles in a slightly older JavaScript application, I wrote an article about that as well. That implementation though, required a redeployment of the total application every time a new user or group was added with a feature toggle, but with this new solution that’s unnecessary.

In this blog, I’ll show you how to add new information to existing feature toggles in a React UI without having to redeploy the whole application.

The best part? It’s not terribly complicated.

NOTE: this article does not eliminate the need to modify and redeploy UI code to implement brand new feature toggles, but it does make it possible to modify existing feature toggles with new information (like new users or user groups).

Rebuilding and redeploying UIs is not always a must

Back in the olden days of JavaScript development, when Grunt and Gulp tasks rebuilt jQuery fueled applications, if you wanted to modify feature toggles in the UI with new information, the whole application had to be rebuilt from the Node.js server up. Recompiled, re-minified, re-uglified, rebuilt and redeployed to the server or the cloud.

It didn’t really take too long in the grand scheme of things, but it was still a pain. And for my business partners who wanted the relative ease of clicking a button to enable or disable a feature toggle for certain user groups, it made that impossible.

<figcaption>This is how easy my business partners want feature toggling to be: just push the button.</figcaption>

</figure>

Now, with newer tools at our disposal like Webpack and hot reloading in the browser with frameworks like React, it doesn’t have to be that way anymore.

You can simply update the information being provided to the UI from the config server, refresh it via the Node.js server, and voilà — the modifications are picked up in the browser.

Feature toggle setup

Enabling feature toggles with the Cloud Config Client npm package

In order to set up our JavaScript application to connect to the Spring Cloud Config server, I used a package called cloud-config-client from npm.

There’s a dozen packages like this made for connecting Node.js applications to Spring Boot cloud configuration servers, but this one is one of the more popular, well-documented packages on npm, so it’s the one I chose to go with this time.

I broke these steps down into server side and client side so you can see where the Node.js files are in play versus the React files.

Server side: configure in `server.js`

After initially installing the cloud-config-client package with npm i --save cloud-config-client, it’s time to configure the Node’s server.js file. The place where every good Express-powered Node application begins.

My server.js file is relatively small, as a lot of the routes, loggers, and other Node server extras have been split off into their own folders to keep it more organized.

The only adjustments needed to this file are requiring the route to the config server at the top of the file:

const configServerRouter = require('./routes/configServer');

And then making sure that when the React client’s request comes to the Node server and includes /configserver/ in the URL path, it routes to the configServer.js file, where the logic to access the Spring Cloud Config server is stored. Like so:

app.use('/configserver', configServerRouter);

server.js

<figcaption>There’s not much to update in the Node’s server.js file to make it able to access the cloud config server.</figcaption>

</figure>

Next, I move on to the configServer.js file, which resides inside my routes/ directory on the server side.

Server Side: Connect to the Spring Cloud and Create The Two Node Config Server Endpoints

The configServer.js file contains the majority of the connection information to the Spring Cloud config server running in the cloud.

Below, is a screenshot of the whole file — I’ll go over each section in detail, but at a high level, it contains a function call to initially connect to the config server and load any feature toggle info that may be present: loadConfig().

There’s an endpoint to refresh the info supplied to the config server with the /refresh endpoint. This is the key to enabling or disabling feature toggles without having to rebuild and redeploy the whole JavaScript application every time.

And finally, there’s the actual /featureToggles/:feature endpoint, which is how the React UI calls to the Node.js server to figure out which feature toggles are available.

Ok, now let’s get in to the details of each function.

configServer.js

<figcaption>Here’s my whole configServer.js file for checking and refreshing feature toggles.</figcaption>

</figure>

Server side: feature toggle endpoint

Here’s a close up of the loadConfig() and /featureToggles/:feature functions that are used to:

Fetch any feature toggle info from the Spring Cloud config server, and
Handle calls from the UI to check if certain feature toggles are available for user groups.

For the actual loadConfig() function, I just followed the documentation supplied by the cloud-config-client package.

Once the initial connection is made from that function, there’s a good bit of Lodash functions being used to massage the config server data into a map of keys and values for each individual feature toggle. If that’s a little hard to follow, it might make more sense when I show an example of the feature toggle in action. For now, keep reading.

The router.get(/featureToggles/:feature) function is the endpoint that will handle the call from the React client, check if that :feature exists as a key in the previously created configServerMap and if it does, return a positive 200 HTTP response to the client side.

That’s all there is to it for this bit.

configServer.js

<figcaption>A close up of both the initial loadConfig() call that gets the info from the Spring Cloud server, and the feature toggle call sent from the client side checking the configServerMap.</figcaption>

</figure>

Server side: feature toggle refresh endpoint

The other function to be aware of here is the router.post(/refresh) endpoint. This endpoint can be called from either the browser, or (more likely) an API REST client like Postman or Insomnia.

This is how changes made to the config server files get updated and pulled into the UI.

This ‘/refresh’ endpoint is the key to modifying the information provided to already existing feature toggles in the React UI.

As you can see, the very same loadConfig() function that’s called as soon as the Node.js server starts up the first time, is called again when the /refresh endpoint is triggered. And to let users know the refresh worked, a simple JSON message is sent back when the POST is successful: res.send("Refreshed Config Server.");.

configServer.js

<figcaption>The refresh endpoint triggered to bring in new information related to feature toggles already implemented in the UI application. This is the key.</figcaption>

</figure>

Client side: feature toggle call

Now, we get to the client side in the React UI, and it’s dead simple. On the UI side of things, I have a folder inside of React called /services where all the shared business logic files live and there’s one file concerned with the config server, appropriately called configServerService.js.

This serves as the connection point between the React UI client and the Node.js server.

As you can probably tell from the screenshot below, this example feature toggle is called "status-pilot" and it checks a user’s ldap (username), which is provided by the Spring Cloud config server through the /configServer/featureToggles/status-pilot endpoint. Then it checks the client supplied ldap against the list of users from the config server and if one of them matches the names in the list, it returns a truth-y boolean.

With this confirmation the user is allowed to see certain features currently being tested out in the app with a small, select group of users. And that’s it.

Just call the isStatusPilotUser() function wherever the React app is checking the user’s permissions in the UI and set the boolean of isStatusPilotUser to true or false after the Node server gives its response based on the info from the config server.

This same sort of function set up can be added to this service file for multiple feature toggles at once, as well.

configServerService.js

<figcaption>This is the function on the client side: isStatusPilotUser() that communicates with the Node server that’s holding feature toggle information provided by the config server.</figcaption>

</figure>

Feature toggles in action

Right, so now that I’ve walked through how to connect the Node server to the Spring Cloud config server, and connect the React client to the Node server, it’s time to show a feature toggle in action.

The first thing needed is a simple: an updated configuration YAML file for the Spring Cloud config server to consume.

Continuing with the "status-pilot" example from above, here’s a YAML file, with a list of LDAPs (usernames). The highlighted username "AM00002" is a new addition to the list that the Node.js server isn’t yet aware of.

This change won’t immediately be registered by the config server itself, I’ll have to trigger it manually to update (or set up a cron job) and get the new information from the file, but I’ll come to that in a minute.

Update the config file for the Spring Cloud config server

Here’s the simple YAML file I mentioned above. The highlighted user "AM00002" is a new addition to the file (and the status-pilot list).

status-pilot-example.yml

<figcaption>Here’s an example YAML file. "AM00002" is a new user who’s been added to the ‘status-pilot’ feature toggle.</figcaption>

</figure>

Verify the config info being sent to the Node server

Currently, when I start up the Node server connected to the Spring Cloud server and the React UI client, it pulls in the previously existing list of info from the Spring Cloud config server. Because the Spring server’s not aware the info it is supplying has changed, it continues to supply the previously cached configuration info to the client.

If you look carefully in the screenshot on the right hand side, you can see that user "AM00002" is not present in the pilotUserList. That’s fine, this will be updated soon.

configServer.js

<figcaption>Just confirming who is and is not currently considered as part of the pilotUserList by the UI’s Node server. Notice "AM00002" is not in that list yet...</figcaption>

</figure>

Hit the Node server’s `/refresh` endpoint with Postman

Time to tell the Spring server to update the files it’s serving. Open up the Postman REST client and type in the URL of your app’s /refresh endpoint and send a POST request to it.

If the refresh trigger is successful, the following success message will appear as the response from the server side in Postman.

<figcaption>This is what a successful call from Postman looks like to the config server endpoint.</figcaption>

</figure>

Check that the React UI now sees "AM00002" as a Status Pilot User

Finally, I can verify the changes have taken effect after refreshing the Spring Cloud server’s info via Postman, by putting a breakpoint in the Chrome DevTools on the client’s isStatusPilotUser() function in the configServerService.js.

When the browser pauses on the breakpoint, I can see the usernames contained in the ldapList supplied to the Node server from the Spring config server. And if you look closely, you can see "am00002" is now there; clear as day.

The refresh to the UI’s feature toggle info worked, and all without having to rebuild or redeploy the JavaScript application. And I’m done, now this user is also part of the status-pilot program in this particular UI feature toggle.

configServerService.js

<figcaption>By using these breakpoints, I can tell the modifications to the status-pilot feature toggle took effect, and now "AM00002" is part of the status-pilot as well.</figcaption>

</figure>

Conclusion

The update to the existing feature toggle "status-pilot" worked, and all without having to rebuild or redeploy the JavaScript application.

With just a few additions to the Node server using the cloud-config-client and a call to a /refresh REST endpoint, the React UI’s existing feature toggles can be modified and updated in short order.

Check back in a few weeks — I’ll be writing about JavaScript, React, IoT, or something else related to web development.

Thanks for reading, I hope you can use this implementation to update your own JavaScript application’s feature toggles, and be able to test new features more easily with a subset of your own users.

References & Further Resources

Wikipedia, feature toggle
Cloud config client, npm
Cloud config client, GitHub
Lodash
Postman REST client
Insomnia REST client

nvm, the Easiest Way to Switch Node Environments on Your Machine in a Flash

Sun, 03 Mar 2019 00:00:00 GMT

<figcaption>nvm — a simple bash script to manage multiple active Node.js versions.</figcaption>

</figure>

Multiple Node environments are a pain to develop locally

I'm sure I’m not alone when I tell you that my current development team owns two different UI applications: one built in AngularJS (the old one) and one built in React (the new one). The two work together to serve up a single user experience, while we slowly migrate over the existing screens and functionality from the old, AngularJS application into the new, React application. The end goal is that the React application will one day host the entire application on its own.

I’m sure I’m also not alone when I tell you that the AngularJS application will ONLY run on Node.js version 9 (it crashes and causes weird bugs if it’s not), and our React application needs Node version 10 or above to take advantage of all the ES6 and beyond features.

And you know what? Switching between Node environments for local development is kind of a pain. It’s not easy, it’s something I forget to do frequently (until I have an unexplained issue during development), and frankly, it’s just not the easiest thing to do on a Mac.

Painfully changing Node versions was my lot in life, until a co-worker clued me in to an awesome tool called Node Version Manager (nvm).

nvm is a local web development game changer. Let me tell you how.

Node Version Manager

What is it?

Node Version Manager is exactly what its name says:

[nvm is a] Simple bash script to manage multiple active node.js versions. —nvm, GitHub

While it doesn’t sound complicated, what nvm can do is awesome. It makes it possible to:

Access every long term support (LTS) version of Node.js from v0.1.14 to the latest version today, which happens to be v.11.10.1, as I write this.
Download any one of those remote LTS versions of Node locally with a simple command.
Set up aliases to switch between different downloaded versions of Node with ease.
Default to automatically use whichever version of Node.js is specified if a .nvmrc file is present in a repo.
And it works with multiple types of shells: Sh, Bash, Zsh, Dash, Ksh, (not Fish though).

As long as you’re fairly comfortable with the terminal, you can use NVM.

Setting up nvm

nvm is relatively easy to set up too — after hearing about its benefits, I was able to use the GiHhub README.md to set it up on my computer in short order.

Step 1: Install nvm

The first step is simplest: just install nvm with the curl or wget command provided in the documentation.

cURL command:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash

Wget command:

wget -qO- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash

Step 1.5: Verify nvm in the command line

Close your current terminal for the install to take effect, open a new terminal window and type:

command -v nvm

If it’s installed you’ll get a message like the screenshot below.

<figcaption>This is the nvm response you want to see. If you get a response like nvm command not found, it’s not ready to be used yet.</figcaption>

</figure>

If the version of nvm shows up in the terminal, you’re ready to go and can move on to the section Using nvm.

If you get an error, you’re like me and need to do a bit more manual installation to set up your shell to point to nvm's home directories. Keep reading.

Step 2: Add the nvm directory paths to your shell profile (when needed)

For me, even after installing nvm using the cURL command, I still got an error message in my terminal when I typed the command -v nvm to verify the installation.

At that point, I jumped down the documentation to the Git Install section which had additional notes on how to add the nvm directory paths to the various shell profiles like ~/.bashrc or ~/.zshrc, in my case, since I prefer to use Zsh.

To edit my .zshrc, I just run the following command in the terminal:

nano .zshrc

Scroll down to the bottom of the file and paste in the following lines.

export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"  # This loads nvm bash_completion

Then type CTRL + X from a Mac, Y to save the changes, and Enter and you should be back where you started in the terminal.

If you want to double check your changes are there, you can run cat .zshrc and scroll down with the down arrow key to check the new nvm scripts are at the bottom of the file.

Once you’re sure it’s there, completely quit the terminal program and reopen it and type command -v nvm again, and you should see this:

<figcaption>Now you’re in business. nvm is installed and ready to be used.</figcaption>

</figure>

Now we’re ready to use nvm.

Using nvm

nvm is super simple to use. Here’s the commands you’ll need to know when you want to work with nvm in the command line.

nvm ls-remote

This shows you all available LTS versions of Node.js.

<figcaption>This is just a fraction of all the available Node.js versions nvm has to offer.</figcaption>

</figure>

nvm ls

Shows you all installed versions available locally on your machine.

<figcaption>The versions of Node.js installed on my local machine.</figcaption>

</figure>

nvm install node OR nvm install 8.15.1

Typing nvm install node will install the latest version of Node.js to your machine, or nvm install <SPECIFIC_NODE_VERSION> will install the specified version.

<figcaption>I just installed the latest version of Node.js to my local development machine.</figcaption>

</figure>

nvm use node OR nvm use 11.10.0

This command will set the version of Node.js running locally to the latest version downloaded if you just type nvm use node, or the version specified if you append the command with the version number nvm use --version e.g. nvm use 8.15.1.

<figcaption>Both of the use commands: the first defaults to the latest version (11.10.1), the second sets the version to 11.10.0.</figcaption>

</figure>

NOTE: The version you want to run locally must be installed on your machine first, or else Node will fall back to whatever version it was running before the command was issued.

nvm run node OR nvm run 11.10.0

This command is almost the same as the one above, the only difference is typing nvm run node or nvm run --version like nvm run 11.10.0 will switch to the specified version of Node.js and open up a Node command line for you to run commands manually from afterwards.

<figcaption>From nvm run node, I can then run commands like console.log(‘hello world’) or server.js right after switching to a Node version.</figcaption>

</figure>

In this way, you could, potentially, have multiple terminals open running multiple versions of Node at once. 🤔 Pretty handy…

nvm alias default node OR nvm alias default 11.10.0

These commands will set an NVM alias called ‘default’ to the latest downloaded version of node with nvm alias default node, or the specified version with nvm alias default --version like nvm alias default 11.10.0.

Once this default alias is set, any new shell will default to running with that version of Node.

</figure>

These are the main commands you’ll probably use to download and switch between Node.js versions while doing local web development. At least, they’re the ones I use most of the time.

The nvm documentation though, is pretty good and it goes into real depth if you want to get fancy with your nvm-ing.

Conclusion

Node versions are something we rarely think about until they become a problem during development. And if your situation is at all similar to mine, you may need to switch between multiple versions regularly, because your various web apps demand it.

Node Version Manager makes it incredibly easy to do this right from the command line. From installation to actual use, nvm is simple, and it makes development in whatever version of Node.js that’s required, so much simpler as well. I hope you’ll take advantage of it.

Thanks for reading, I hope nvm can be useful to you and your development team when you’re writing your various JavaScript applications.

References & Further Resources

nvm, GitHub

How ESLint Makes Me a Better React Developer

Sun, 17 Feb 2019 00:00:00 GMT

</figure>

Clean-er code: the wake up call

I was a little late to the React party, I only started learning it in earnest, about 10 months ago. Yes, I’d heard about it long before then. Yes, I’d had co-workers, casual acquaintances, and the two developers putting out my web dev podcast of choice, Syntax, tell me how great it was. Yes, I’d heard Facebook, which at that time had yet to reveal of its less-than-savory-doings, had created it, and kept improving on it.

But as anyone who’s been a web developer for longer than six months knows, there’s an absolute ton of JavaScript frameworks out there that had a brief moment in the sun, only to be obscured weeks or month later by the new hotness. So I sat on my hands, waited to see if React was worth investing my time in learning, and laughed every time this meme came across my Internet feeds.

<figcaption>This image still makes me smile whenever I see it. O’Reilly parody covers are the best.</figcaption>

</figure>

When it became apparent that React was the new JavaScript powerhouse of choice and wasn’t going away anytime soon, I finally got on board and started teaching it to myself. For the past 10 months or so, I’ve been learning React on the side while supporting an AngularJS 1.5 application at my current company.

And having to work with that monolithic, two-way data binding, AngularJS application, chock full of business logic, while building smaller, cleaner, unidirectional data flow, React side projects reinforced what every one else had been telling me all along. React is way better.

Recently, I was feeling pretty good about my own React / JavaScript knowledge and coding abilities. By no means am I an expert, nor have I used some of React’s most recent advancements like Hooks, but I can talk knowledgeably about state and props. I can discuss different lifecycle render methods and what they’re good for. Heck, I can even debate Redux versus the Context API for state management, and why things like defined prop types are beneficial.

I’ve written a full-stack, user registration app as a learning tool and I am proud of it. It’s React and ES6 on the front end, Node.js on the backend, and a MySQL database with the Sequelize ORM to handle database interactions. I implemented Passport.js middleware for JWT authentication and Docker Compose to spin the whole application up at once. I also added in password reset via email functionality with Nodemailer and Swagger API endpoint testing.

I’ve done a lot with this little app, learned a lot along the way, and written a lot of articles about my learnings (all linked above). But then, I added ESLint into my application, thinking it would be a good way to make sure my code was well written...

And once again, I was humbled and taught how to write better JavaScript code. That’s what I want to talk about today: how ESLint can make me (and you) a better developer.

ESLint: 21st century rules for ES6 JavaScript & beyond

</figure>

What is ESLint?

If you’re not familiar with ESLint, you can read this handy post I wrote a few months ago. The post was more to do with using both ESLint and Prettier in the Visual Studio Code IDE to produce consistent, clean code across development teams with little effort on the part of the developers, but it still explains how ESLint works.

In a nutshell:

Linting tools like ESLint allow developers to discover problems with their JavaScript code without executing it. —ESLint site

ESLint is a file in a project repo called .eslintrc with a list of linting rules that runs through the project’s JavaScript code and finds problematic patterns or code that don’t adhere to certain style guidelines set forth by those rules. Then it alerts developers so they can fix it the errors.

It will not (normally) rewrite or reformat code like Prettier. Nor will it tell you if you have bad business logic. But it will tell you if curly braces are missing, return statements are unreachable, or objects / arrays could be destructured, among other things.

Why ESLint is awesome

This is awesome because not only does ESLint identify ways to make code better, but if you don’t need or don’t agree with certain rules, they can be changed or ignored (either for the line, for the whole file, or for the whole project).

None of the rules depend on each other, they all function independently, and some rules can even fix the code themselves to fall in line with the prescribed rules.

Airbnb ESLint

The ESLint rules that Airbnb abides by are considered among many as the gold standard for React applications. They are strict, they are unforgiving and they are thorough. They are so popular, in fact, that Airbnb created their ESLint config as an npm package that has over 1 million weekly downloads, as I write this article.

They also have an entire JavaScript style guide on GitHub that has some very good advice in it for writing code. But that’s a different rabbit hole than the one I’m going down today.

Since I felt like holding myself to high standards (being a masochist), I decided to use Airbnb’s ESLint rules as the starting place for my own .eslintrc file for my full stack JavaScript project. Boy, did I learn a lot when I turned on the linter...

Before and after ESLint's cold, brutally honest linting

My ESLint setup & dev dependencies

Before I could get linting for my React application going to reveal my own code’s shortcomings, I had to download a few dev dependencies for my package.json file and set up my .eslintrc file.

Dev dependencies

I used Create React App to create the initial front end for my full stack application, so I only had a few additional dependencies I needed to add to my package.json file to make Airbnb’s ESLint config work.

npm install --save-dev eslint eslint-config-airbnb eslint-plugin-import eslint-plugin-jsx-a11y eslint-plugin-react

.eslintrc file

Then, once all the package were installed, I set up my .eslintrc file like this.

{  
  "extends": "airbnb",  
  "parser": "babel-eslint",  
  "env": 
    {    
      "node": true,    
      "es6": true,    
      "browser": true  
    },  
  "rules": 
    {    
      "react/jsx-filename-extension": [1, { "extensions": [".js", ".jsx"] }],    
      "implicit-arrow-linebreak": "off",    
      "comma-dangle": "off",    
      "indent": "off",    
      "no-trailing-spaces": "off"  
    }
  }

That’s really all the setup I did. I added "extends": "airbnb" as prescribed by the Airbnb’s documentation, added in that I was transpiling my ES6 code with babel, listed the environments and added a few additional rules to override the Airbnb settings.

Most of these rules were overwritten because I also use Prettier’s code formatting on save in my VS Code setup, and if I didn’t turn off the rules in ESLint, Prettier would continue to reformat code and break the rules every time I saved a file. A minor inconvenience though.

The rules I set

I extended the react/jsx-filename-extension rule to apply to files ending in both .js and .jsx because all my files which include JSX end in .js.
I turned off implict-arrow-linebreak because Prettier reformatted it on save.
I turned off comma-dangle, indent, and no-trailing-spaces for the same reason.

Beyond that, there were one-off rules that ESLint highlighted, which I turned off by file or by line when I needed to (such as console.log). I’m sure you’ll also come upon situations that dictate such exceptions as well.

Without ESLint enabled

Now, let’s see how my code looked before I enabled ESLint in my project.

Here’s an example of two of my React components. The first is the home page file, the second is the header bar, which is a reusable component across the whole application.

Home.js file

<figcaption>This code looks pretty nice and unassuming right? Nothing particularly wrong with it.</figcaption>

</figure>

This file looks OK at first glance, it seems clean and well formatted. Simple, straightforward. One prop that gets passed to the header bar itself, styling for the buttons comes from other reusable components. All pretty self-explanatory.

HeaderBar.js file

<figcaption>Again, this code looks OK to me...</figcaption>

</figure>

The header bar file is much the same. Nice, well known React create class component structure, some props for the header title, a default to fall back to if one wasn’t provided. Seems fine to me.

With ESLint enabled

Then, I turned on ESLint in my project (all the details on how to do this in VS Code are here), and I saw the following horrors on pretty much every file in my project…

One of the many things I appreciate about the VS Code IDE is that it has its own separate “Problems” tab where all of these ESLint errors show up. And when I hover over all the angry red squiggles or click on any of the errors, I’m directed right in the code to where those errors are. Presumably to fix them (or ignore them if necessary).

Clicking the little lightbulbs that appear over the angry code will show you what ESLint rule is being broken and give you the option to ignore the rule for the line or whole file or go to the ESLint rule documentation to learn more about how to fix the error.

Home.js file

<figcaption>😱 ESLint does not feel the same way about this code that I did. All the things are wrong.</figcaption>

</figure>

Um…what is going on here? Why are there so many red squigglies everywhere?!?!

Header.js file

<figcaption>Ugh, look at all those red lines...</figcaption>

</figure>

😱 And here too? Really?!? Such a simple component, and this is what I see? What gives ESLint??

After I’d gotten over my initial shock and indignation at ESLint’s very brutal and unforgiving code review, I set to work fixing the errors and learned how to write better React in the process.

I did it through a combination of reading the ESLint error documentation (which VS Code’s ESLint lightbulb extension linked me right to) and checking Stack Overflow for examples of how to write things the right way based on the rule being broken.

Fixed after ESLint

Let’s see what the new pro-ESLint-ed code looks like for the same files.

Home.js file

<figcaption>😆 Much nicer. Look — it’s clean, it’s pure and stateless, it’s easier to read.</figcaption>

</figure>

Here’s what changed.

ESLint detected this home file was actually a pure, stateless component that didn’t need to be a React class. Instead, it could be written as a const variable: const Home.
It also identified that the <LinkButtons> didn’t need curly braces around the buttonText and link properties since both properties were defined as string data types prop types in the LinkButtons.js component file. Here’s what that looks like.

<figcaption>Here’s the prop types for LinkButtons that wasn’t previously defined before ESLint was in play.</figcaption>

</figure>

You’ll also notice in the code snippet above, the ESLint rule I disabled regarding buttonStyle: // eslint-disable-next-line react/forbid-prop-types. I did that because ESLint frowns on prop types described only as object (it wants greater specificity), but since I was using the button styles provided by my React Material styling dependency, I chose not to go deeper in defining the prop types beyond object in my own prop types.

Happily, with these changes, the problems and red squiggles displayed in VS Code, were gone, for this file. And I have to admit, the code does look cleaner than my original React code.

HeaderBar.js file

<figcaption>Again, looks better. Props and default props are defined, the component is stateless, title is destructured from props object being passed in.</figcaption>

</figure>

Here’s the header bar reusable component rewritten after ESLint as well.

This component too, could be written as a stateless, functional component, instead of a React class. It became const HeaderBar.
ESLint instructed me to use ES6 object destructuring to take just the title property from the object that I was passing in to the headerBar component.
I had not defined propTypes or defaultProps for the component, the kind of typechecking that helps prevent bugs from unnecessarily entering code just because an unexpected JavaScript type was passed as a prop.

After these changes were made, my code was deemed acceptable by ESLint’s strict standards.

Now, imagine if you will, a series of similar errors in pretty much all the files in my code base. But these errors helped me learn how to recognize and write better JavaScript and React code, and for that I am grateful.

My project was small enough I could go through file by file and fix the issues in just a few sessions, but if are implementing ESLint into a large, already existing code base, it might make more sense to just fix the files you’re currently working on as you implement new features, until all the files can be fixed to adhere to the new ESLint rules. Instead of trying to fix them all in one fell swoop.

Conclusion

JavaScript is an interpreted language, and it allows developers a lot of leeway that strongly typed languages like Java won’t put up with. Most of the time, I appreciate this flexibility, but it does open up opportunities for bugs strongly typed languages don’t have to deal with.

As usual, when I started to feel too confident in my JavaScript abilities, ESLint came along and took me down a few notches. But the harshness of the Airbnb linter I view as a good thing.

It taught me how to better recognize and refactor React classes into stateless components, how to take advantage of ES6 features like object destructuring, and how defining prop types can prevent unnecessary bugs before they happen. It has improved my code and my coding, and will continue to do so in the future.

Check back in a few weeks, I’ll be writing about React or something else related to web development.

Thanks for reading, I hope this helps convince you to give ESLint a try with your own React and JavaScript projects and improve your own projects and coding abilities.

References & Further Resources

GitHub repo of MERN application
ESLint Documentation
Airbnb ESLint, npm
Airbnb ESLint, GitHub
Airbnb JavaScript Style Guide, GitHub
Syntax podcast

The Case for Standardized Error Handling in Your Web Application’s APIs

Sun, 03 Feb 2019 00:00:00 GMT

</figure>

Introduction

Bad error messages are a sin we’re all guilty of as programmers. On the server side, on the client side, they are rampant basically everywhere in web development.

I write them. You write them. My whole team writes them. We all agree they’re bad: the error messages don’t tell us or our users what the real problem is and only through digging deep into the codebase and tracing through application logs can we locate the actual culprit — most of the time.

This is not a new or unusual phenomenon. It’s standard, and it makes everyone’s lives infinitely more painful than they need to be. But every time we talk about how to make the error messaging better — more verbose, more pinpointed, less nebulous about what actual error occurred, these efforts to improve error messaging get deprioritized by whoever has the power to direct the dev team and we move on to more straightforward, end-user-benefitting features.

And the vicious cycle continues. A user encounters an error on client side of the application, the development team is contacted after the error can’t be resolved by turning the system off and on again (see meme below), the devs look through tons of logs and try to reproduce error locally (which is sometimes an impossible feat), the team either reproduces the error and figures out what caused it and how to fix it, or they can’t reproduce the error and close the bug with no fix, until it rears its ugly head again in the future.

<figcaption>I couldn’t resist the opportunity to use this meme — it’s so accurate sometimes it’s frightening.</figcaption>

</figure>

Does this sound like an optimum system to you? Me neither. Which is why I’m here, writing to you today.

I’m making the case for standardizing error handling from APIs so we can all stop wasting so much time tracking down bugs, and get back to the coding we all want to do — building cool things that make people’s lives better.

Standardized errors: the need & the ground rules

<figcaption>A standard, totally unhelpful, API error message example.</figcaption>

</figure>

In case you didn’t read the error message above, the part that matters says: Failed to execute ImportKey action in set service. Deleting created cart item. : 500 Internal Server Error.

Yes, this is an actual error message from one of my team’s actual backend web applications. No, it doesn’t make any sort of sense; I totally agree with you.

In web development, we place a lot of emphasis on the “happy path”. The path that we want and expect our users to take when they’re using our tool or application — it’s the path we code for, the path we optimize for, the path we place most of our focus on.

The “unhappy path”, is the path when a user doesn’t do the right thing, click the right button, or generally figures out how to use the system in some way other than its intended purpose. This is the path where things go wrong, and with bad error messaging (like the above), we, as both users and developers, have a really hard time figuring out what exactly went wrong. And in general, the unhappy paths (and the errors they produce) are given a lot less thought about how they should be handled in an effective manner.

That needs to change. Here’s what I propose in the name of better error handling, all of which is based on the OData v4 JSON specification.

What is OData?

</figure>

Before I give my recommendations, let me give a little background on OData.

OData stands for Open Data Protocol, and it is an open protocol which allows the creation and consumption of queryable and interoperable RESTful APIs in a simple and standard way. Microsoft initiated OData in 2007. — Wikipedia, OData

In essence, OData defines a set of best practices for building and consuming RESTful APIs (application program interface). These practices help developers focus on business logic while building RESTful APIs without having to worry about the various approaches to define things like request and response headers, status codes, HTTP methods, URL conventions, media types, payload formats, query options and more.

It proposes standards to make web development a little less haphazard and a little more predictable, regardless of programming language or development approach and team.

Now, we know about OData and its legitimacy and reason for being in the world, let’s get down to the rules of handling errors.

8 error rules to follow

Below are the standards my team and wider company are working to enact with our applications — they’re generic enough recommendations to apply broadly to APIs in general, but specific enough to remove the questions around how to go about implementing the recommendations.

Rule 1: Code one, reusable error handler

For nonsuccess conditions, developers should be able to write one piece of code that handles errors consistently across different REST API methods.
This allows for the building of simple and reliable infrastructure to handle exceptions as a separate flow from successful responses.
Keep in mind, this error handler is very generic and does not require specific OData constructs. APIs should use this format even if they are not using other OData constructs.

Rule 2: The JSON error handler must have an error object

The error response must be a single JSON object. This object must have a name/value pair named error. The value must be a JSON object.

<figcaption>Example of the most basic error object.</figcaption>

</figure>

Rule 3: error must contain code and message, and may contain other properties For more information

This object must contain name/value pairs with the names code and message and it may contain name/value pairs with the names target, details, and innererror.
The value for the code name/value pair is a language-independent string. Its value is a service-defined error code that should be human-readable.
This code serves as a more specific indicator of the error than the HTTP error code specified in the response.
API Methods should have a relatively small number (about 20) of possible values for code and all clients must be capable of handling all of them.
Most services will require a much larger number of more specific error codes, which are not interesting to all clients. These error codes should be exposed in the innererror name/value pair as described below.
Introducing a new value for code that is visible to existing clients is a breaking change and requires a version increase. API Methods can avoid breaking changes by adding new error codes to innererror instead.

<figcaption>Example of the error object with multiple properties included.</figcaption>

</figure>

Rule 4: The message should help humans debug the error

The value for the message name/value pair must be a human-readable representation of the error. It is intended as an aid to developers and is not suitable for exposure to end users.
API methods wanting to expose a suitable message for end users must do so through an annotation or custom property.
API methods should not localize message for the end user, because doing so may make the value unreadable to the app developer who may be logging the value, as well as make the value less searchable on the Internet.

Rule 5: target is the name of the property in error

The value for the target name/value pair is the target of the particular error (e.g., the name of the property in error).

Rule 6: details is an array of objects with code and message

The value for the details name/value pair must be an array of JSON objects that must contain name/value pairs for code and message, and may contain a name/value pair for target as described above. The objects in the details array usually represents distinct, related errors that occurred during the request. See example below.

Example of the details object:

{   
  "error": {     
    "code": "BadArgument",     
    "message": "Multiple errors in ContactInfo data",     
    "target": "ContactInfo",     
    "details": [       
      {         
        "code": "NullValue",         
        "target": "PhoneNumber",         
        "message": "Phone number must not be null"       
      },       
      {         
        "code": "NullValue",         
        "target": "LastName",         
        "message": "Last name must not be null"       
      },       
      {         
        "code": "MalformedValue",         
        "target": "Address",         
        "message": "Address is not valid"       
       }     
     ]   
   } 
 }

In this example there were multiple problems with the request, with each of the individual errors listed in details.

Rule 7: innererror is an object with service-defined contents

The value for the innererror name/value pair must be an object.
The contents of this object are service-defined. API methods wanting to return more specific errors than the root-level code must do so by including a name/value pair for code and a nested innererror.
Each nested innererror object represents a higher level of detail than its parent.
When evaluating errors, clients must traverse through all of the nested innererrors and choose the deepest one that they understand. This scheme allows services to introduce new error codes anywhere in the hierarchy without breaking backwards compatibility, so long as old error codes still appear.
The service may return different levels of depth and detail to different callers. For example, in development environments, the deepest innererror may contain internal information that can help debug the service.
To guard against potential security concerns around information disclosure, services should take care not to expose too much detail unintentionally.
Error objects may also include custom server-defined name/value pairs that may be specific to the code. Error types with custom server-defined properties should be declared in the service’s metadata document. See example below.
Error responses may contain annotations in any of their JSON objects.

<figcaption>Example of innererror object.</figcaption>

</figure>

Example of innererror object:

{   
  "error": {     
    "code": "BadArgument",     
    "message": "Previous passwords may not be reused",     
    "target": "password",     
    "innererror": {       
      "code": "PasswordError",       
      "innererror": {         
        "code": "PasswordDoesNotMeetPolicy",         
        "minLength": "6",         
        "maxLength": "64",         
        "characterTypes":   ["lowerCase","upperCase","number","symbol"],         
        "minDistinctCharacterTypes": "2",         
        "innererror": {           
          "code": "PasswordReuseNotAllowed"         
        }       
      }     
    }   
  } 
}

In the example above, the most basic error code is BadArgument, but for clients that are interested, there are more specific error codes in innererror.

The PasswordReuseNotAllowed code may have been added by the service at a later date, having previously only returned PasswordDoesNotMeetPolicy.

Existing clients do not break when the new error code is added, but new clients may take advantage of it. The PasswordDoesNotMeetPolicy error also includes additional name/value pairs that allow the client to determine the server’s configuration, validate the user’s input programmatically, or present the server’s constraints to the user within the client’s own localized messaging.

Rule 8: Code for failure with retries

It is recommended that for any transient errors that may be retried, services should include a Retry-After HTTP header indicating the minimum number of seconds that clients should wait before attempting the operation again.

And this concludes the rules to a standardized API error handling strategy. It’s not overly complicated, but it recommends good guidelines to try to adhere to when encountering the unhappy path.

Conclusion

Error messaging isn’t the most fun thing to implement, but it’s even less fun to try and debug a painfully vague (and oftentimes misleading) server message. Standardized error handling is a must — especially if you’re part of a larger team of developers, or providing an API service to external clients.

Organizations like OData have helped establish best practices around writing and using APIs, which helps developers focus on the business logic of their applications instead. By following these best practices, we’ve established a set of easy to understand and repeatable rules for error handling to make the whole process of dealing with the inevitable unhappy path simpler.

Check back in a few weeks, I’ll be writing about React or something else related to web development, so please follow me so you don’t miss out.

Thanks for reading, I hope this helps you more effectively handle errors in a standardized format so you can better catch and fix bugs.

References & Further Resources

Setting Up Swagger to API Test In a JavaScript Application

Mon, 21 Jan 2019 00:00:00 GMT

</figure>

Introduction

If you’re like me, and you’ve ever worked with a REST API (application program interface) that doesn’t have a UI attached to it, you’ve probably stumbled more than once wondering what inputs and auth tokens the API needs to give you back the data you’re expecting and not some cryptic 400 HTTP error message.

Sometimes I’ve got no idea if I need some kind of authorization header (or how to send it), if the API will accept POSTs or PUTs (and what parameters are required versus what’s optional), and if it wants application/json or some other XML formatted encoding instead.

Wouldn’t it be nice to be to take some of that uncertainty out of the equation? Wouldn’t it be nice if, instead of being faced with an empty Postman or Insomnia API testing application waiting for you to configure it and provide it with everything it needs, an API route told you exactly what it needed? Request body vs. query parameter, integer vs. string, JWT auth header vs. cookie — all of it, just laid out, all the possible routes visible.

Sounds too good to be true? It’s not.

Before I joined my current team of developers, I had no idea such a magical API-assistant like this existed, but it does. And it’s called Swagger.

Today, we'll cover how to set up Swagger inside of a JavaScript or Node.js application and document the actual code so that you can see exactly what REST API routes are available to you and what they need to work in the browser.

I’ll give you all the details below, but here’s an example of what Swagger can provide for your JavaScript web application’s API routes.

<figcaption>Yes, your API too can be this lovely, alphabetized and color-coded with Swagger’s help.</figcaption>

</figure>

Meet Swagger, your API's new friend

Swagger’s Wikipedia page sums up what it is best:

Swagger is an open-source software framework backed by a large ecosystem of tools that helps developers design, build, document, and consume RESTful Web services. While most users identify Swagger by the Swagger UI tool, the Swagger toolset includes support for automated documentation, code generation, and test-case generation. — Wikipedia, Swagger (software)

If this doesn’t make total sense now, read on. It will make sense once you see it in practice.

One of the loveliest things about Swagger is the fact that it can be used with most any programming language: from JavaScript to Haskell to Go to C++ to Python to Rust, the list goes on and on. The amount of supported languages is really staggering.

My first introduction to Swagger was through Java’s Spring Boot applications. By including one line in the build.gradle dependencies: compile(‘io.springfox:springfox-swagger-ui:2.9.2’), one annotation in the Java config files: @EnableSwagger2, and the location of the swagger.basePackage in the application.yml, you’re provided with a beautiful API UI when the Spring Boot application compiles and runs.

The Swagger UI can be reached by typing in something like http://localhost:8081/swagger-ui.html into the browser. Then, voilà, you’ve got an interactive, browser-based UI for your API service that you can test to your heart’s content.

<figcaption>Here’s an example of what I see when I open up a Spring Boot Swagger HTML page.</figcaption>

</figure>

Swagger presents a user with the routes in the project, it shows exactly what the shape of the data is and what types of properties are required for each parameter, it lets you know if headers are needed and exactly what kinds they are, if a request body or query param is optional or required, as well as possible HTTP response methods and what each one means. It’s amazing — all that for just three little updates to my Java code.

It seemed like a piece of cake with Spring Boot, and I (naively) thought it would be just as easy to implement Swagger with my full stack, MERN, JavaScript application. Silly me…

Swagger in JavaScript, there's an npm package for that, right?

Well, the short answer to that question is: yes. The longer answer to that question is: there are many packages. The longest answer is what I’ll tell you now.

Unlike Spring Boot, which has, what could be considered a defacto Swagger package, the world of JavaScript and Swagger is more fractured and fragmented. From my exploration of npm and the Internet, there didn’t appear to be just one answer to generate Swagger docs and display them in the UI format I’d come to know and love.

So I read some blogs, read some documentation around OpenAPI specification (formerly known as Swagger Specification), and upon learning there was no maintained Swagger docs generator for JavaScript, I ultimately decided I would use Swagger JSDoc to create my Swagger documentation for my routes, and combine it with Swagger UI Express to generate the UI interface in the browser. Now let me tell you a bit more about each of these modules.

Swagger JSDoc

Once again, Swagger JSDoc’s own words best describe it:

swagger-jsdoc enables you to integrate Swagger using JSDoc comments in your code. Just add @swagger on top of your DocBlock and declare the meaning of your code in YAML complying to the OpenAPI specification. — Swagger JSDoc, npm

This means that above each existing API route, you write in your Swagger specs YAML-style (so indentation and colons count) for that route with the @swagger annotation, and this module will then translate that into the UI rendered by my second module: Swagger UI Express.

This module is also especially good for already written APIs (which my project happens to be). I was adding in Swagger documentation as an afterthought — not as part of the initial architecture when I first built the app.

the swagger-jsdoc project assumes that you want document your existing/living/working code in a way to "give life" to it, generating a specification which can then be fed into other Swagger tools, and not the vice-versa.— Swagger JSDoc, npm

If I had decided to add Swagger earlier in the process, I would have looked at Swagger Editor or Swagger Node.

Swagger UI Express

Swagger UI Express's documentation is much less verbose and involved than Swagger JSDoc, because the challenging part really is the writing of the Swagger specifications.

As Swagger UI Express’s docs say, it:

Adds middleware to your express app to serve the Swagger UI bound to your Swagger document. This acts as living documentation for your API hosted from within your app. — Swagger UI Express, npm

Basically, you follow a prescribed formula to feed the Swagger documentation into the Swagger UI Express module and it generates the HTML UI. And thankfully, the documentation for how to do this is both up to date and accurate. It makes it really simple.

Now that I’ve given a little more background on the two solutions I chose to make Swagger work with JavaScript, and more specifically, my Express/Node.js server application, it’s time to talk about how to implement it.

Let's set up Swagger in JavaScript, how hard can it be?

Once I got the hang of how to write the route specifications, implementing Swagger wasn’t too tough, but to start out with the easiest part, I’ll go over the Swagger UI Express and Swagger JSDoc server setup first.

If you’d like to download my whole project and run it locally or just see the source code, here’s a link to the repo or you can click the link of each file name above a code snippet to see it in GitHub.

Server.js setup

server.js

// more JS imports here

import swaggerJSDoc from 'swagger-jsdoc';
import swaggerUi from 'swagger-ui-express';

const app = express();

const API_PORT = process.env.API_PORT || 3000;

const swaggerDefinition = {
  info: {
    title: 'MySQL Registration Swagger API',
    version: '1.0.0',
    description: 'Endpoints to test the user registration routes',
  },
  host: 'localhost:3003',
  basePath: '/',
  securityDefinitions: {
    bearerAuth: {
      type: 'apiKey',
      name: 'Authorization',
      scheme: 'bearer',
      in: 'header',
    },
  },
};

const options = {
  swaggerDefinition,
  apis: ['./routes/*.js'],
};

const swaggerSpec = swaggerJSDoc(options);

app.get('/swagger.json', (req, res) => {
  res.setHeader('Content-Type', 'application/json');
  res.send(swaggerSpec);
});

app.use('/api-docs', swaggerUi.serve, swaggerUi.setup(swaggerSpec));

// more JS code here

As you can see, both swaggerJSDoc and swaggerUiExpress are imported into the file (this can also be done with require if you choose not to use an ES6 transpiler in Node.js like I did).

This is where the magic of Swagger UI Express transforms Swagger JSDoc’s swaggerSpec into a nice browser UI.

The swaggerDefinition object is created (this is part of the setup required by swagger-jsdoc). Its properties include info like a title for the app, a version and a description. It also needs a host (in my case, port 3003 is the port exposed from my Docker container for the API) and a basePath. Since I also have some routes that are only accessible to the user if they’re logged in with a valid JWT token, I also had to create one securityDefinitions object with bearerAuth info: type, name, scheme, and in.

Here is the swaggerDefinition code:

const swaggerDefinition = {
  info: {
    title: 'MySQL Registration Swagger API',
    version: '1.0.0',
    description: 'Endpoints to test the user registration routes',
  },
  host: 'localhost:3003',
  basePath: '/',
  securityDefinitions: {
    bearerAuth: {
      type: 'apiKey',
      name: 'Authorization',
      scheme: 'bearer',
      in: 'header',
    },
  },
};

Once that’s defined, it’s gathered together into another object called options, which includes the path for your route files. In my case, I have a folder called routes where they all live, so that’s where I told Swagger to look.

const options = {
  swaggerDefinition,
  apis: ['./routes/*.js'],
};

After that, the swaggerSpec variable is initialized, which wraps the options object in swaggerJSDoc(). Then, the Express server is given a route to see the full swagger.json schema that gets created by the swaggerSpec.

const swaggerSpec = swaggerJSDoc(options);

app.get('/swagger.json', function(req, res) {
  res.setHeader('Content-Type', 'application/json');
  res.send(swaggerSpec);
});

But to me, this is secondary to the piece of the code, below, which is what actually combines the swaggerSpec with the swaggerUi module that creates the Swagger UI I know and love.

app.use('/api-docs', swaggerUi.serve, swaggerUi.setup(swaggerSpec));

Right, and that’s that. It was a bit more setup than I initially thought, but as I said, the documentation for setting all this up is decent, and if you get stuck, there’s tutorials like this one to help you out.

User schema

Before I could write my routes, referencing my User object, I needed to go to the actual object declaration file and tell Swagger what made up my user. After that, I’d be able to reference that schema and those defining properties in the different routes to ensure things were copacetic.

I used Sequelize as the ORM for my MySQL database, so I had a User schema defined for that reason. If you’d like to learn more about Sequelize, you can read this blog post I wrote.

Here’s what my user.js file ended up looking like. Everything between the

/**
 * @swagger 
 * schema definition info goes here...
 */

is part of the schema definition that will be referenced in my routes.

user.js

/* eslint-disable indent */
/**
 * @swagger
 * definitions:
 *   User:
 *     type: object
 *     properties:
 *       id:
 *         type: integer
 *       first_name:
 *         type: string
 *       last_name:
 *         type: integer
 *       email:
 *         type: string
 *       username:
 *         type: string
 *       password:
 *         type: string
 *         format: password
 *       resetPasswordToken:
 *         type: string
 *       resetPasswordExpires:
 *         type: string
 *         format: date-time
 *       required:
 *         - email
 *         - username
 *         - password
 */

module.exports = (sequelize, type) => sequelize.define('user', {
    id: {
      type: type.INTEGER,
      primaryKey: true,
      autoIncrement: true,
    },
    first_name: type.STRING,
    last_name: type.STRING,
    email: {
      type: type.STRING,
      allowNull: false,
    },
    username: {
      type: type.STRING,
      allowNull: false,
    },
    password: {
      type: type.STRING,
      allowNull: false,
    },
    resetPasswordToken: type.STRING,
    resetPasswordExpires: type.DATE,
  });

As you can see, each of the object’s properties is named ( id, username, resetPasswordToken, etc.), and its data type ( string, integer, etc.) is defined right underneath it. The last thing of note is the required object properties.

For me, these match the actual User object’s properties that have the requirement allowNull: false. That is username, password, and email for my app.

Route schema

The next thing to cover is writing the Swagger documentation for each of the routes.

I actually have nine different routes in my application, but I’ll be showing the documentation for three.

One will be a route requiring no authorization, the second will be one requiring a JWT token for authorization, and the third one will need a different randomly generated string to verify it is legit. I’ll explain that one in more detail when I get there.

loginUser.js - the unsecured route

/* eslint-disable no-console */
import jwt from 'jsonwebtoken';
import passport from 'passport';
import jwtSecret from '../config/jwtConfig';
import User from '../sequelize';

/**
 * @swagger
 * /loginUser:
 *   post:
 *     tags:
 *       - Users
 *     name: Login
 *     summary: Logs in a user
 *     produces:
 *       - application/json
 *     consumes:
 *       - application/json
 *     parameters:
 *       - name: body
 *         in: body
 *         schema:
 *           $ref: '#/definitions/User'
 *           type: object
 *           properties:
 *             username:
 *               type: string
 *             password:
 *               type: string
 *               format: password
 *         required:
 *           - username
 *           - password
 *     responses:
 *       '200':
 *         description: User found and logged in successfully
 *       '401':
 *         description: Bad username, not found in db
 *       '403':
 *         description: Username and password don't match
 */

module.exports = app => {
  app.post('/loginUser', (req, res, next) => {
    passport.authenticate('login', (err, users, info) => {
      if (err) {
        console.error(`error ${err}`);
      }
      if (info !== undefined) {
        console.error(info.message);
        if (info.message === 'bad username') {
          res.status(401).send(info.message);
        } else {
          res.status(403).send(info.message);
        }
      } else {
        req.logIn(users, () => {
          User.findOne({
            where: {
              username: req.body.username,
            },
          }).then(user => {
            const token = jwt.sign({ id: user.id }, jwtSecret.secret, {
              expiresIn: 60 * 60,
            });
            res.status(200).send({
              auth: true,
              token,
              message: 'user found & logged in',
            });
          });
        });
      }
    })(req, res, next);
  });
};

My first Swagger documentation example is for an unsecured route: my user login route, to be precise. Once again, all the Swagger documentation is written in this fashion, and be warned: spacing/indentation and colons do matter.

/**
 * @swagger
 * documentation for route goes here...
 */

Swagger documentation is written in YAML syntax, so it cares about proper indentation for each line and it also cares that there are colons for most lines — if the spacing is off or the colon is missing, the terminal running the Node server code will throw errors until it’s fixed (this tripped me up a few times as I was going through this exercise).

Here’s the full Swagger documentation for the loginUser() route.

/**
 * @swagger
 * /loginUser:
 *   post:
 *     tags:
 *       - Users
 *     name: Login
 *     summary: Logs in a user
 *     produces:
 *       - application/json
 *     consumes:
 *       - application/json
 *     parameters:
 *       - name: body
 *         in: body
 *         schema:
 *           $ref: '#/definitions/User'
 *           type: object
 *           properties:
 *             username:
 *               type: string
 *             password:
 *               type: string
 *               format: password
 *         required:
 *           - username
 *           - password
 *     responses:
 *       '200':
 *         description: User found and logged in successfully
 *       '401':
 *         description: Bad username, not found in db
 *       '403':
 *         description: Username and password don't match
 */

As you can see, the route is defined first (the actual URL route Swagger will have to hit when it’s run in the browser), then the type of HTTP call is defined ( get, post, delete).

The summary section appears next to the route in the UI, and it nicely describes what the route is used for.

If there were numerous different schemas defined, the tags section would be more useful in organizing different route groups. Since I only have users it doesn’t really matter much for me.

In case there’s something besides JSON being produced or consumed, that is defined.

And then the parameters that the user will input are laid out underneath. Since my JSON is being passed in a request body, in: body is defined. If it were being passed in the query params it would say in: query instead.

The only two properties of the user object which is defined and referenced with $ref: '#/definitions/User' are username and password. Since both are required for this route to work the required field is set underneath and both properties are referenced (note that must be inline with the schema column for the required fields to take effect in Swagger).

Finally, the HTTP responses we can expect from the server are laid out with descriptions of what each one means. They’re pretty self-explanatory.

Now that I’ve covered an unsecured route, let’s move on to a JWT secured route.

findUser.js - the secured route

/* eslint-disable no-console */
import passport from 'passport';
import User from '../sequelize';

/**
 * @swagger
 * /findUser:
 *   get:
 *     tags:
 *       - Users
 *     name: Find user
 *     summary: Finds a user
 *     security:
 *       - bearerAuth: []
 *     consumes:
 *       - application/json
 *     produces:
 *       - application/json
 *     parameters:
 *       - in: query
 *         name: username
 *         schema:
 *           type: string
 *         required:
 *           - username
 *     responses:
 *       '200':
 *         description: A single user object
 *         schema:
 *           $ref: '#/definitions/User'
 *       '401':
 *         description: No auth token / no user found in db with that name
 *       '403':
 *         description: JWT token and username from client don't match
 */

module.exports = (app) => {
  app.get('/findUser', (req, res, next) => {
    passport.authenticate('jwt', { session: false }, (err, user, info) => {
      if (err) {
        console.log(err);
      }
      if (info !== undefined) {
        console.log(info.message);
        res.status(401).send(info.message);
      } else if (user.username === req.query.username) {
        User.findOne({
          where: {
            username: req.query.username,
          },
        }).then((userInfo) => {
          if (userInfo != null) {
            console.log('user found in db from findUsers');
            res.status(200).send({
              auth: true,
              first_name: userInfo.first_name,
              last_name: userInfo.last_name,
              email: userInfo.email,
              username: userInfo.username,
              password: userInfo.password,
              message: 'user found in db',
            });
          } else {
            console.error('no user exists in db with that username');
            res.status(401).send('no user exists in db with that username');
          }
        });
      } else {
        console.error('jwt id and username do not match');
        res.status(403).send('username and jwt token do not match');
      }
    })(req, res, next);
  });
};

This route is my findUser route, it gets triggered after the user has logged in and the profile page component starts to mount on the client side to gather all the user info from the database.

For this route’s Swagger documentation, once again, it defines the URL path, the post type, the tags, summary, produces / consumes specs, params, etc.

What’s different is that this route includes the security field, which is where the securityDefinitions I set up in the server.js file come back into play. I am passing a JWT (JSON Web Token) back and forth between my client and server to verify a user’s permissions and authorization, so by including

/**
 * @swagger
 * ...
 * security:
 *   - bearerAuth: []
 * ...
 */

it secures the route and requires a user to include a JWT before successfully making the call. bearerAuth is just what I named the object in the server.js file, it can be named however you please, the two just have to match in the server and route files, so Swagger knows how to form the request correctly.

Once the JWT token is passed in and the request is sent, the Passport.js middleware I have picks up the authorization header, decodes the JWT token and verifies it’s good. If you’d like to learn more about setting up Passport with a React / Node.js project, you can read my blog post here.

Here’s the code snippet of the Swagger documentation, I want to point out a couple more things about it that differ from the previous route.

/**
 * @swagger
 * /findUser:
 *   get:
 *     tags:
 *       - Users
 *     name: Find user
 *     summary: Finds a user
 *     security:
 *       - bearerAuth: []
 *     consumes:
 *       - application/json
 *     produces:
 *       - application/json
 *     parameters:
 *       - in: query
 *         name: username
 *         schema:
 *           type: string
 *         required:
 *           - username
 *     responses:
 *       '200':
 *         description: A single user object
 *         schema:
 *           $ref: '#/definitions/User'
 *       '401':
 *         description: No auth token / no user found in db with that name
 *       '403':
 *         description: JWT token and username from client don't match
 */

Aside from the security option, the other difference is in the parameters. As I mentioned in the previous route, if the user input is coming through the request’s query parameters, instead of the request body, the parameters look just a smidge different.

Here’s the piece to focus on — under parameters the -in field becomes query instead of body. This tells Swagger to pass whatever properties are needed in the query params of the request.

/** 
 * @swagger
 * ...
 * parameters:
 *   - in: query
 * ...
 */

The other thing to notice is in the responses section at the bottom of the documentation. For the 200 HTTP response, it actually produces an application/json object that holds all the user information. I need to reference the User schema so that Swagger understands how to map the response object it receives without having to write out all the object properties again.

/** 
* @swagger 
* ...
*     responses:
*       200:
*         description: A single user object
*         schema:
*           $ref: '#/definitions/User' 
* ...
*/

Good, now I’ve detailed a secure route and an unsecured route with Swagger, now it’s time for a third, slightly different route. One that coincides with a password reset email.

resetPassword.js - the randomly generated string route

/* eslint-disable no-console */
/* eslint-disable max-len */
import Sequelize from 'sequelize';
import User from '../sequelize';

// eslint-disable-next-line prefer-destructuring
const Op = Sequelize.Op;

/**
 * @swagger
 * /reset:
 *   get:
 *     tags:
 *       - Users
 *     name: Reset Password Link
 *     summary: Create validation string in reset password link to verify user's allowed to reset their password
 *     consumes:
 *       - application/json
 *     parameters:
 *       - name: resetPasswordToken
 *         in: query
 *         schema:
 *           type: string
 *         required:
 *           - resetPasswordToken
 *     responses:
 *       '200':
 *         description: User's password reset link is valid
 *       '403':
 *         description: Password reset link is invalid or has expired
 */

module.exports = (app) => {
  app.get('/reset', (req, res) => {
    User.findOne({
      where: {
        resetPasswordToken: req.query.resetPasswordToken,
        resetPasswordExpires: {
          [Op.gt]: Date.now(),
        },
      },
    }).then((user) => {
      if (user == null) {
        console.error('password reset link is invalid or has expired');
        res.status(403).send('password reset link is invalid or has expired');
      } else {
        res.status(200).send({
          username: user.username,
          message: 'password reset link a-ok',
        });
      }
    });
  });
};

This is the last route example I’ll cover here, if you’d like to see all my routes (which include all of the basic CRUD-functionality and then some), you can see my GitHub repo.

I include this because this route actually requires a non-expired, randomly generated string which is sent in the body of an email to a user if they’ve forgotten their login password. It’s a little different and a little tricky, so I thought it was worth going over.

If you’d like to read more about sending password reset emails using Node, you can see this blog I wrote.

Here’s the code snippet of the Swagger documentation below.

/**
 * @swagger
 * /reset:
 *   get:
 *     tags:
 *       - Users
 *     name: Reset Password Link
 *     summary: Create validation string in reset password link to verify user's allowed to reset their password
 *     consumes:
 *       - application/json
 *     parameters:
 *       - name: resetPasswordToken
 *         in: query
 *         schema:
 *           type: string
 *         required:
 *           - resetPasswordToken
 *     responses:
 *       '200':
 *         description: User's password reset link is valid
 *       '403':
 *         description: Password reset link is invalid or has expired
 */

There’s really not any big differences that you can see, but you will notice, under the parameters the in references a query param agin, meaning that the random string attached the URL link that’s sent to a user’s email address will need to be included to verify the password reset is authorized.

I’ll show this one in the testing section, and it should make more sense there.

Summing it up

Now, you have seen how to setup both swagger-jsdoc and swagger-ui-express in a JavaScript project. I modified the main server.js file, added a data model to the User object schema and wrote Swagger documentation for three separate Node.js routes.

To get to this point, I ended up referencing the OpenAPI Specification docs a lot and the documentation around both npm modules. Don’t be afraid to use either — they’re actually pretty darn good, and there’s a ton of stuff you can add to make your Swagger docs even more helpful.

Ok, that’s finished. Let’s move on to testing the API in the browser.

Testing Swagger in the browser

Testing the API in the browser via Swagger was actually a very good exercise for me, because it uncovered some logic flaws I was making in my code, and it forced me to rethink how I was handling certain situations (like sending error codes and messages from the server to the client).

So, first things first, fire up the whole application or the API individually, whichever you choose. I like to just load the whole thing, because I wrote a single docker-compose.yml file which takes care of it all by writing docker-compose up in the terminal after the initial docker-compose build.

Once the API gets up and running, you can access the in-browser interface, by putting this URL in: http://localhost:3003/api-docs/ — at least, that’s the external port my API’s Docker container is running on, and the / api-docs/ URL path is what I specified in the server.js Swagger UI Express setup.

app.use('/api-docs', swaggerUi.serve, swaggerUi.setup(swaggerSpec));

If you used something different, reference that URL route instead.

Now, if all goes according to plan, you should see a screen like this.

<figcaption>This is what my Swagger interface looks like when I first hit the URL.</figcaption>

</figure>

At this point, you can click any of the links and test out your API endpoints. I’m going to test my /loginUser endpoint first.

So I click that post, click the “Try It Out” button in the top right of the expanded view, and then replace the two "string" placeholders with a valid username and password (which would be created in the database using the /registerUser route), and hit the big blue "Execute" button below.

<figcaption>Filling out the request body (note the "*required" next to the body).</figcaption>

</figure>

As the image above shows our request body inputs are required ("*required"), so if they’re left as empty string, you’ll get back a 403 "Missing Credentials" response from the API.

All the response codes I’ve written out in the Swagger documentation are there to decipher what just happened. Here’s what a successful, 200 HTTP login response looks like.

<figcaption>A successful, 200 HTTP response.</figcaption>

</figure>

When a valid username and password are entered in, which you can see from the cURL request box which shows you exactly what the call was to the API (helpful when debugging calls that aren’t working), the /loginUser route gives back a response body object with a token property that is the JWT token that will be stored in the client’s local storage and passed back in the authorization headers for every secured route call.

I’ll show now how to test one of those routes. I’ll use the /findUser route. First, copy the value from the token property returned from the login endpoint, then open up the /findUser input, and click the little lock in the top right hand of the input box. This will open up a modal for you to input the JWT token string.

<figcaption>JWT authorization modal.</figcaption>

</figure>

In here in the "Value" input box enter JWT <token string>.

<figcaption>Add in JWT <token string> in the input box and hit the "Authorize" button.</figcaption>

</figure>

For my particular application, the Authorization header that goes with each request, needs the JWT ahead of the actual JWT string. That reference is how Passport finds the correct header string to decode and verify if the user’s authorized. It must be included in this format for this API call to work. After this is done, click the "Authorize" button, then the little lock icon will be locked for this route and you can fill out the user info and test your call.

This is what a 200 HTTP response on this secured route will look like.

<figcaption>A successful call to the secured route using a JWT token in the "Authorization" header. </figcaption>

</figure>

The cURL request box shows the route with the username query parameter and it shows the authorization JWT token I entered in the lock on the route, which is needed to return the User object. Without the JWT token, or with an invalid token the 401 or 403 HTTP responses would be sent instead.

The final request I’ll show is a Swagger request that deals with a password reset email (which I wrote about implementing here). That one’s a bit fiddly because the URL link that’s sent to the user’s email address to reset the password contains the query string needed to verify its validity AND it’s only good for the next 10 minutes.

So, if a user was to send a password reset email to their inbox, they’d receive an email with the link like the image below.

<figcaption>The reset email the user receives — the random string at the end is what will be needed for Swagger.</figcaption>

</figure>

From here, the user would need to copy the string after the /reset/ and paste that into the swagger /reset route within 10 minutes of it being sent. This randomly generated string has a timestamp attached to it, and part of the authorization involves checking the string and the timestamp against the database.

If both are valid for that user, here’s what a successful verification through Swagger would look like.

<figcaption>The successful 200 response to verify the password reset token for this user is valid, which allows them to reach the secured route to reset their password.</figcaption>

</figure>

If the string is invalid or the link had expired, instead Swagger would show this message.

<figcaption>The 403 error a user would get if s/he were to submit a password reset token that was either invalid or had expired.</figcaption>

</figure>

And that, my friends, is how you can test your API using Swagger’s handy interface in the browser.

Conclusion

This is how to implement Swagger in your JavaScript or Node.js application for in-browser API testing. The initial setup isn’t quite as easy as Spring Boot makes it, but for an interpreted language like JavaScript, it really wasn’t that hard, once I got the hang of it.

Check back in a few weeks — I’ll be writing more about JavaScript, React, or something else related to web development.

Thanks for reading, I hope this helps you more easily document and test your own web apps.

References & Further Resources

MERN JavaScript app repo
Swagger site documentation
Swagger, Wikipedia article
OpenAPI Specifications (Swagger Specification)
Swagger JSDoc, npm
Swagger UI Express, npm
YAML syntax reference

Using Java to Read Really, Really Large Files

Fri, 04 Jan 2019 00:00:00 GMT

</figure>

Introduction

Anyone who knows me fairly well in my programming life will know that I’m not as partial to Java.

I’m a JavaScript developer first and foremost. It was what I learned first, it confounded then delighted me after I started to get the hang of it, and it made a heck of a lot more sense to me than Java, what with its compilation, its need to declare every single variable type (yes, I know the latest versions of Java have done away with this requirement for some of the simpler inferences), and its massive libraries of maps, lists, collections, etc.: HashMaps, Maps, HashTables, TreeMaps, ArrayLists, LinkedLists, Arrays, it goes on and on.

That being said, I do make an effort to get better at Java and the other shortcomings I have by not having a traditional computer science degree. If you’d like to know more about my atypical path to becoming a software engineer, you can read my very first blog post here.

So when I told a colleague of mine about a coding challenge I’d come across a couple months back and how I solved it (and then performance tested my various solutions) in JavaScript, he looked back at me and said: “How would you solve it using Java?”

<figcaption>My initial reaction to the question of using Java.</figcaption>

</figure>

I stared at him as the wheels began to turn in my head, and I accepted the challenge to find an optimal solution in Java, as well.

In this post, I'll show you a variety of solutions (and their performance) to parse through really, really large datasets using the Java programming language.

The challenge I faced

So before I get down to how I solved my challenge using Java, let me actually recap the requirements.

In the original article I wrote on using Node.js to read really, really large files, this was a coding challenge issued to a developer for an insurance technology company.

The challenge was straightforward enough: download this large zip file of text from the Federal Elections Commission, read that data out of the .txt file supplied, and supply the following info:

Write a program that will print out the total number of lines in the file.
Notice that the 8th column contains a person’s name. Write a program that loads in this data and creates an array with all name strings. Print out the 432nd and 43243rd names.
Notice that the 5th column contains a form of date. Count how many donations occurred in each month and print out the results.
Notice that the 8th column contains a person’s name. Create an array with each first name. Identify the most common first name in the data and how many times it occurs.

Link to the data: https://www.fec.gov/files/bulk-downloads/2018/indiv18.zip*

NOTE: I add the asterisk after the file link because others who have chosen to undertake this challenge themselves have actually seen the file size increase since I downloaded it back in the beginning of October of 2018. At last count, someone mentioned it was up to 3.5GB now, so it seems as if this data is still live and being added to all the time. I believe the solutions I present below will still work though, but your counts and numbers will vary from mine.

I liked the challenge and wanted some practice manipulating files, so I decided to see if I could figure it out.

Now, without further ado, let’s talk about some different solutions I came up with to read really, really large files in Java.

The three Java-based solutions I came up with

Java has long been a standard programming language with file processing capabilities, and as such, there’s been a large number of ever improving ways to read, write and manipulate files with it.

Some methods are baked straight into the core Java framework, and some are still independent libraries that need to be imported and bundled together to run. But regardless, I came up with three different methods to read files in Java, and then I performance tested them to see which methods were more efficient.

Below are snippets of various pieces of code, if you’d like to see all of the original code, you can access my Github repo here.

You’ll notice that I used the same code logic to extract the data from each file, the major differences are between the initial file connection and text parsing. I did this so that I could have a more accurate idea of how the different methods stacked up against each other for the performance part of my evaluation. Apples to apples comparisons and all that.

Solution #1: Java `FileInputStream()` and `Scanner()` Implementation

The first solution I came up with uses Java’s built-in FileInputStream() method combined with Scanner().

In essence, FileInputStream just opens the connection to the file to be read, be it images, characters, etc. It doesn’t particularly care what the file actually is, because Java reads the input stream as raw bytes of data. Another option (at least for my case) is to use FileReader() which is specifically for reading streams of characters, but I went with FileInputStream() for this particular scenario. I used FileReader() in another solution I tested later on.

Once the connection to the file is established, Scanner comes into play to actually parse the text bytes into strings of readable data. Scanner() breaks its inputs into tokens using a delimiter pattern, which by default matches whitespace (but can also be overridden to use regex or other values). Then, by using the Scanner.hasNextLine() boolean, and the Scanner.nextLine() method, I can read the contents of the text file line by line and pull out the pieces of data that I need.

Scanner.nextLine() actually advances this scanner past the current line and returns the input that was skipped, which is how I’m able to gather the required info from each line until there’s no more lines to read and Scanner.hasNextLine() returns false and the while loop ends.

Here’s a sample of code using FileInputStream() and Scanner().

File f = new File("src/main/resources/config/test.txt");

try {
   FileInputStream inputStream = new FileInputStream(f);
   Scanner sc = new Scanner(inputStream, "UTF-8");
   // do some things ...
   while (sc.hasNextLine()) {
      String line = sc.nextLine();
     //  do some more things ...
   } 
  // do some final things
}

And here is my full code to solve all the tasks laid out above.

NOTE: Click the title of any of these files to go to GitHub and see the full working code repo.

ReadFileJavaApplicationFileInputStream.java

package com.example.readFile.readFileJava;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.HashMap;
import java.util.LinkedList;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Scanner;

public class ReadFileJavaApplicationFileInputStream {

	public static void main(String[] args) throws IOException {
    File f = new File(Common.getPathToTargetFile(args));

    try {
      FileInputStream inputStream = new FileInputStream(f);
      Scanner sc = new Scanner(inputStream, "UTF-8");

      // get total line count
      Instant lineCountStart = Instant.now();
      int lines = 0;

      Instant namesStart = Instant.now();
      ArrayList<String> names = new ArrayList<>();

      // get the 432nd and 43243 names
      ArrayList<Integer> indexes = new ArrayList<>();

      indexes.add(1);
      indexes.add(433);
      indexes.add(43244);

      // count the number of donations by month
      Instant donationsStart = Instant.now();
      ArrayList<String> dates = new ArrayList<>();

      // count the occurrences of first name
      Instant commonNameStart = Instant.now();
      ArrayList<String> firstNames = new ArrayList<>();


      System.out.println("Reading file using File Input Stream");

      while (sc.hasNextLine()) {
        String line = sc.nextLine();
        lines++;

        // get all the names
        String array1[] = line.split("\\s*\\|\\s*");
        String name = array1[7];
        names.add(name);
        if (indexes.contains(lines)) {
          System.out.println("Name: " + names.get(lines - 1) + " at index: " + (lines - 1));
        }

        if (name.contains(", ")) {

          String array2[] = (name.split(", "));
          String firstHalfOfName = array2[1].trim();

          if (!firstHalfOfName.isEmpty()) {
            if (firstHalfOfName.contains(" ")) {
              String array3[] = firstHalfOfName.split(" ");
              String firstName = array3[0].trim();
              firstNames.add(firstName);
            } else {
              firstNames.add(firstHalfOfName);
            }
          }
        }

        String rawDate = array1[4];
        String month = rawDate.substring(4, 6);
        String year = rawDate.substring(0, 4);
        String formattedDate = month + "-" + year;
        dates.add(formattedDate);

      }
      sc.close();

      Instant namesEnd = Instant.now();
      long timeElapsedNames = Duration.between(namesStart, namesEnd).toMillis();
      System.out.println("Name time: " + timeElapsedNames + "ms");

      System.out.println("Total file line count: " + lines);
      Instant lineCountEnd = Instant.now();
      long timeElapsedLineCount = Duration.between(lineCountStart, lineCountEnd).toMillis();
      System.out.println("Line count time: " + timeElapsedLineCount + "ms");

      HashMap<String, Integer> dateMap = new HashMap<>();
      for (String date : dates) {
        Integer count = dateMap.get(date);
        if (count == null) {
          dateMap.put(date, 1);
        } else {
          dateMap.put(date, count + 1);
        }
      }
      for (Map.Entry<String, Integer> entry : dateMap.entrySet()) {
        String key = entry.getKey();
        Integer value = entry.getValue();
        System.out.println("Donations per month and year: " + key + " and donation count: " + value);

      }
      Instant donationsEnd = Instant.now();
      long timeElapsedDonations = Duration.between(donationsStart, donationsEnd).toMillis();
      System.out.println("Donations time: " + timeElapsedDonations + "ms");

      HashMap<String, Integer> map = new HashMap<>();
      for (String name : firstNames) {
        Integer count = map.get(name);
        if (count == null) {
          map.put(name, 1);
        } else {
          map.put(name, count + 1);
        }
      }

      LinkedList<Entry<String, Integer>> list = new LinkedList<>(map.entrySet());

      Collections.sort(list, new Comparator<Map.Entry<String, Integer>>() {
        public int compare(Map.Entry<String, Integer> o1,
                            Map.Entry<String, Integer> o2) {
          return (o2.getValue()).compareTo(o1.getValue());
        }
      });
      System.out.println("The most common first name is: " + list.get(0).getKey() + " and it occurs: " + list.get(0).getValue() + " times.");
      Instant commonNameEnd = Instant.now();
      long timeElapsedCommonName = Duration.between(commonNameStart, commonNameEnd).toMillis();
      System.out.println("Most common name time: " + timeElapsedCommonName + "ms");

    } catch (FileNotFoundException e) {
      e.printStackTrace();
    }
	}
}

After the file’s data is being read one line at a time, it’s simply a matter of getting the necessary data and manipulating it to fit my needs.

Task #1: Get the file’s total line count

Getting the line count for the entire file was easy. All that involved was a new int lines = 0 declared outside the while loop, which I incremented each time the loop ran again.

Request #1: done.

Task #2: Create a list of all the names and find the 432nd and 43243rd names

The second request, which was to collect all the names and print the 432nd and 43243rd names from the array, required me to create an ArrayList<String> names = new ArrayList<>(); and an ArrayList<Integers> indexes = new ArrayList<>(); which I promptly added the indexes of 432 and 43243 with indexes.add(433) and indexes.add(43244), respectively.

I had to add 1 to each index to get the correct name position in the array because I incremented my line count (starting at 0) as soon as Scanner.hasNextLine() returned true. After Scanner.nextLine() returned the previous line’s contents I could pull out the names I needed, which meant its true index (starting from index 0) was actually the index of the line count minus one. (Trust me, I triple checked this to make sure I was doing my math correctly).

I used an ArrayList for the names because it maintains the element's insertion order which means while displaying ArrayList elements the result set will always have the same order in which the elements got inserted into the List. Since I’m iterating through the file line by line, the elements will always be inserted into the list in the same order.

Here’s the full logic I used to get all the names and then print out the names if the indexes I had in my indexes ArrayList matched the lines count index.

int lines = 0;
ArrayList<String> names = new ArrayList<>();

// get the 432nd and 43243 names
ArrayList<Integer> indexes = new ArrayList<>();

indexes.add(433);
indexes.add(43244);

System.out.println("Reading file using File Input Stream");

while (sc.hasNextLine()) {
   String line = sc.nextLine();
   lines++;
   // get all the names
   String array1[] = line.split("\\s*\\|\\s*");
   String name = array1[7];
   names.add(name);
   if (indexes.contains(lines)) {
      System.out.println("Name: " + names.get(lines - 1) + " at
      index: " + (lines - 1));
   } 
   // ...
}

Request #2: done.

Task #3: Count how many donations occurred in each month

As I approached the donation counting request, I wanted to do more than count donations by month, I wanted to count them by both month and year, as I had donations from both 2017 and 2018.

The very first thing I did was set up an initial ArrayList to hold all my dates: ArrayList<String> dates = new ArrayList<>();.

Then, I took the 5th element in each line, the raw date, and used the substring() method to pull out just the month and year for each donation.

I reformatted each date into easier-to-read dates and added them to the new dates ArrayList.

String rawDate = array1[4];
String month = rawDate.substring(4, 6);
String year = rawDate.substring(0, 4);
String formattedDate = month + "-" + year;
dates.add(formattedDate);

After I’d collected all the dates, I created a HashMap to hold my dates: HashMap<String, Integer> dateMap = new HashMap<>();, and then looped through the dates list to either add the dates as keys to the HashMap if they didn’t already exist or increment their value count, if they did exist.

Once the HashMap was made, I ran that new map through another for loop to get each object’s key and value to print out to the console. Voilà.

The date results were not sorted in any particular order, but they could be by transforming the HashMap back in to an ArrayList or LinkedList, if need be. I chose not to though, because it was not a requirement.

HashMap<String, Integer> dateMap = new HashMap<>();
for (String date : dates) {
   Integer count = dateMap.get(date);
   if (count == null) {
      dateMap.put(date, 1);
   } else {
      dateMap.put(date, count + 1);
   }
}
for (Map.Entry<String, Integer> entry : dateMap.entrySet()) {
   String key = entry.getKey();
   Integer value = entry.getValue();
   System.out.println("Donations per month and year: " +
   entry.getKey() + " and donation count: " + entry.getValue());
}

Request #3: done.

Task #4: Identify the most common first name and how often it occurs

The fourth request, to get all the first names only and find the count of the most commonly occurring one, was trickiest.

It required me to first, check if the names array contained a comma (there were some business names that had no commas), then split() the name on the comma and trim() any extraneous white space from it.

Once that was cleaned up, I had to check if the first half of the name had any white spaces (meaning the person had a first name and middle name or possibly a moniker like “Ms.”) and if it did, split() it again, and trim() up the first element of the newly made array (which I presumed would almost always be the first name).

If the first half of the name didn’t have a space, it was added to the firstNames ArrayList as is. That’s how I collected all the first names from the file. See the code snippet below.

// count the occurrences of first name
ArrayList<String> firstNames = new ArrayList<>();

System.out.println("Reading file using File Input Stream");

    while (sc.hasNextLine()) {
        String line = sc.nextLine();

        // get all the names
        String array1[] = line.split("\\s*\\|\\s*");
        String name = array1[7];
        names.add(name);

        if (name.contains(", ")) {
            String array2[] = (name.split(", "));
            String firstHalfOfName = array2[1].trim();
  
            if (firstHalfOfName != undefined ||
                !firstHalfOfName.isEmpty()) {
                     if (firstHalfOfName.contains(" ")) {
                       String array3[] = firstHalfOfName.split(" ");
                       String firstName = array3[0].trim();
                       firstNames.add(firstName);
                     } else {
                       firstNames.add(firstHalfOfName);
                     }
                  }
               }

Once I’ve collected all the first names I can, and the while loop reading the file has ended, it’s time to sort the names and find the most common one.

For this, I created another new HashMap: HashMap<String, Integer> map = new HashMap<>();, then looped through all the names and if the name didn’t exist in the map already, it was created as the map’s key and the value was set as 1. If the name already existed in the HashMap, the value was incremented by 1.

HashMap<String, Integer> map = new HashMap<>();
for (String name : firstNames) {
   Integer count = map.get(name);
   if (count == null) {
      map.put(name, 1);
   } else {
      map.put(name, count + 1);
   }
}

But wait — there’s more! Once we have the HashMap, which is unordered by nature, it needs to be sorted from largest to smallest value to get the most commonly occurring first name, so I transform each entry in the HashMap into a LinkedList, which can be ordered and iterated through.

LinkedList<Entry<String, Integer>> list = new LinkedList<>(map.entrySet());

And finally, the list is sorted using the Collections.sort() method, and invoking the Comparator interface to sort the name objects according to their value counts in descending order (highest value is first). Check this out.

Collections.sort(list, new Comparator<Map.Entry<String, Integer>>()
  {
     public int compare(Map.Entry<String, Integer> o1,
     Map.Entry<String, Integer> o2) {
         return (o2.getValue()).compareTo(o1.getValue());
    }
  }
);

Once all of that has been done, the first key value pair of the LinkedList can finally be pulled out and displayed to the user. Here’s the whole shebang that’s put together once the all the first names have been read out of the file.

HashMap<String, Integer> map = new HashMap<>();
for (String name : firstNames) {
   Integer count = map.get(name);
   if (count == null) {
      map.put(name, 1);
   } else {
      map.put(name, count + 1);
   }
}

LinkedList<Entry<String, Integer>> list = new LinkedList<>(map.entrySet());

Collections.sort(list, new Comparator<Map.Entry<String, Integer>>()
  {
     public int compare(Map.Entry<String, Integer> o1,
     Map.Entry<String, Integer> o2) {
         return (o2.getValue()).compareTo(o1.getValue());
    }
  }
);

System.out.println("The most common first name is: " + list.get(0).getKey() + " and it occurs: " + list.get(0).getValue() + " times.");

Request #4 (and arguably the most complicated of all the tasks): done.

Great, now that I’ve given you the soliloquy of my brain’s logic in Java, I can give you much quicker overviews of the other two methods I tried for reading the text data from the files. (Because the logic portion of the code is exactly the same.)

Solution #2: Java `BufferedReader()` and `FileReader()` implementation

My second solution involved two more of Java’s core methods: BufferedReader() and FileReader().

BufferedReader reads text from a character-input stream, buffering characters so as to provide for the efficient reading of characters, arrays, and lines, and it is wrapped around the FileReader method, which is the actual method reading the specified text file. The BufferedReader makes the FileReader more efficient in its operation, that’s all.

BufferedReader’s method readLine() is what actually reads back each line of the text as it is read from the stream, allowing us to pull out the data needed.

The setup is similar to FileInputStream and Scanner; you can see how to implement BufferedReader and FileReader below.

File f = new File("src/main/resources/config/test.txt");

try (BufferedReader b = new BufferedReader(new FileReader(f))) { 
   String readLine = "";
   // do some things ...
   while ((readLine = b.readLine()) != null) { 
     // do some more things...
   } 
  // do some final things
}

And here is my full code using BufferedReader() and FileReader().

ReadFileJavaApplicationBufferedReader.java

package com.example.readFile.readFileJava;

import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.HashMap;
import java.util.LinkedList;
import java.util.Map;
import java.util.Map.Entry;

public class ReadFileJavaApplicationBufferedReader {

	public static void main(String[] args) {
		try {
			File f = new File(Common.getPathToTargetFile(args));

			try (BufferedReader b = new BufferedReader(new FileReader(f))) {
				String readLine = "";

				// get total line count
				Instant lineCountStart = Instant.now();
				int lines = 0;

				Instant namesStart = Instant.now();
				ArrayList<String> names = new ArrayList<>();

				// get the 432nd and 43243 names
				ArrayList<Integer> indexes = new ArrayList<>();

				indexes.add(1);
				indexes.add(433);
				indexes.add(43244);

				// count the number of donations by month
				Instant donationsStart = Instant.now();
				ArrayList<String> dates = new ArrayList<>();

				// count the occurrences of first name
				Instant commonNameStart = Instant.now();
				ArrayList<String> firstNames = new ArrayList<>();


				System.out.println("Reading file using Buffered Reader");

				while ((readLine = b.readLine()) != null) {
					lines++;

					// get all the names
					String array1[] = readLine.split("\\s*\\|\\s*");
					String name = array1[7];
					names.add(name);
					if(indexes.contains(lines)){
						System.out.println("Name: " + names.get(lines - 1) + " at index: " + (lines - 1));
					}

					if(name.contains(", ")) {

						String array2[] = (name.split(", "));
						String firstHalfOfName = array2[1].trim();

						if (!firstHalfOfName.isEmpty()) {
							if (firstHalfOfName.contains(" ")) {
								String array3[] = firstHalfOfName.split(" ");
								String firstName = array3[0].trim();
								firstNames.add(firstName);
							} else {
								firstNames.add(firstHalfOfName);
							}
						}
					}

					String rawDate = array1[4];
					String month = rawDate.substring(4,6);
					String year = rawDate.substring(0,4);
					String formattedDate = month + "-" + year;
					dates.add(formattedDate);

				}

				Instant namesEnd = Instant.now();
				long timeElapsedNames = Duration.between(namesStart, namesEnd).toMillis();
				System.out.println("Name time: " + timeElapsedNames + "ms");

				System.out.println("Total file line count: " + lines);
				Instant lineCountEnd = Instant.now();
				long timeElapsedLineCount = Duration.between(lineCountStart, lineCountEnd).toMillis();
				System.out.println("Line count time: " + timeElapsedLineCount + "ms");

				HashMap<String, Integer> dateMap = new HashMap<>();
				for(String date:dates){
					Integer count = dateMap.get(date);
					if (count == null) {
						dateMap.put(date, 1);
					} else {
						dateMap.put(date, count + 1);
					}
				}
				for (Map.Entry<String, Integer> entry : dateMap.entrySet()) {
					String key = entry.getKey();
					Integer value = entry.getValue();
					System.out.println("Donations per month and year: " + key + " and donation count: " + value);

				}
				Instant donationsEnd = Instant.now();
				long timeElapsedDonations = Duration.between(donationsStart, donationsEnd).toMillis();
				System.out.println("Donations time: " + timeElapsedDonations + "ms");

				HashMap<String, Integer> map = new HashMap<>();
				for(String name:firstNames){
					Integer count = map.get(name);
					if (count == null) {
						map.put(name, 1);
					} else {
						map.put(name, count + 1);
					}
				}

				LinkedList<Entry<String, Integer>> list = new LinkedList<>(map.entrySet());

				Collections.sort(list, new Comparator<Map.Entry<String, Integer> >() {
					public int compare(Map.Entry<String, Integer> o1,
					                   Map.Entry<String, Integer> o2)
					{
						return (o2.getValue()).compareTo(o1.getValue());
					}
				});
				System.out.println("The most common first name is: " + list.get(0).getKey() + " and it occurs: " + list.get(0).getValue() + " times.");
				Instant commonNameEnd = Instant.now();
				long timeElapsedCommonName = Duration.between(commonNameStart, commonNameEnd).toMillis();
				System.out.println("Most common name time: " + timeElapsedCommonName + "ms");
			}

		} catch (IOException e) {
			e.printStackTrace();
		}
	}
}

Besides the BufferedReader and FileReader implementation, though, all the logic within is the same, so I’ll move on now to my final Java file reader implementation: FileUtils.LineIterator.

Apache Commons IO `FileUtils.LineIterator()` Implementation

<figcaption>Apache is responsible for this handy library.</figcaption>

</figure>

The last solution I came up with involves a library made by Apache, called FileUtils.LineIterator(). It’s easy enough to include the dependency. I used Gradle for my Java project, so all I had to do was include the commons-io library in my build.gradle file.

build.gradle

dependencies {
	compile group: 'commons-io', name: 'commons-io', version: '2.6'
}

The LineIterator, does exactly what its name suggests: it holds a reference to an open Reader (like FileReader in my last solution), and iterates over each line in the file. And it’s really easy to set up LineIterator in the first place.

LineIterator has a built-in method called nextLine(), which actually returns the next line in the wrapped reader (not unlike Scanner’s nextLine() method or BufferedReader’s readLine() method).

Here’s the code to set up FileUtils.LineIterator() once the dependency library has been included.

File f = new File("src/main/resources/config/test.txt");

try {
   LineIterator it = FileUtils.lineIterator(f, "UTF-8"); 
   // do some things ...
   while (it.hasNext()) {
      String line = it.nextLine(); 
     // do some other things ...
   }
   // do some final things
}

And here is my full code using FileUtils.LineIterator().

ReadFileJavaApplicationLineIterator.java

package com.example.readFile.readFileJava;

import org.apache.commons.io.FileUtils;
import org.apache.commons.io.LineIterator;

import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.HashMap;
import java.util.LinkedList;
import java.util.Map;
import java.util.Map.Entry;

public class ReadFileJavaApplicationLineIterator {
	public static void main(String[] args) throws IOException {
		File f = new File(Common.getPathToTargetFile(args));

		try {
			LineIterator it = FileUtils.lineIterator(f, "UTF-8");

			// get total line count
			Instant lineCountStart = Instant.now();
			int lines = 0;

			Instant namesStart = Instant.now();
			ArrayList<String> names = new ArrayList<>();

			// get the 432nd and 43243 names
			ArrayList<Integer> indexes = new ArrayList<>();

			indexes.add(1);
			indexes.add(433);
			indexes.add(43244);

			// count the number of donations by month
			Instant donationsStart = Instant.now();
			ArrayList<String> dates = new ArrayList<>();

			// count the occurrences of first name
			Instant commonNameStart = Instant.now();
			ArrayList<String> firstNames = new ArrayList<>();


			System.out.println("Reading file using Line Iterator");

			while (it.hasNext()) {
				String line = it.nextLine();
				lines++;

				// get all the names
				String array1[] = line.split("\\s*\\|\\s*");
				String name = array1[7];
				names.add(name);
				if (indexes.contains(lines)) {
					System.out.println("Name: " + names.get(lines - 1) + " at index: " + (lines - 1));
				}

				if (name.contains(", ")) {

					String array2[] = (name.split(", "));
					String firstHalfOfName = array2[1].trim();

					if (!firstHalfOfName.isEmpty()) {
						if (firstHalfOfName.contains(" ")) {
							String array3[] = firstHalfOfName.split(" ");
							String firstName = array3[0].trim();
							firstNames.add(firstName);
						} else {
							firstNames.add(firstHalfOfName);
						}
					}
				}

				String rawDate = array1[4];
				String month = rawDate.substring(4, 6);
				String year = rawDate.substring(0, 4);
				String formattedDate = month + "-" + year;
				dates.add(formattedDate);
			}

			Instant namesEnd = Instant.now();
			long timeElapsedNames = Duration.between(namesStart, namesEnd).toMillis();
			System.out.println("Name time: " + timeElapsedNames + "ms");

			System.out.println("Total file line count: " + lines);
			Instant lineCountEnd = Instant.now();
			long timeElapsedLineCount = Duration.between(lineCountStart, lineCountEnd).toMillis();
			System.out.println("Line count time: " + timeElapsedLineCount + "ms");

			HashMap<String, Integer> dateMap = new HashMap<>();
			for (String date : dates) {
				Integer count = dateMap.get(date);
				if (count == null) {
					dateMap.put(date, 1);
				} else {
					dateMap.put(date, count + 1);
				}
			}

			for (Map.Entry<String, Integer> entry : dateMap.entrySet()) {
				String key = entry.getKey();
				Integer value = entry.getValue();
				System.out.println("Donations per month and year: " + key + " and donation count: " + value);
			}

			Instant donationsEnd = Instant.now();
			long timeElapsedDonations = Duration.between(donationsStart, donationsEnd).toMillis();
			System.out.println("Donations time: " + timeElapsedDonations + "ms");

			HashMap<String, Integer> map = new HashMap<>();
			for (String name : firstNames) {
				Integer count = map.get(name);
				if (count == null) {
					map.put(name, 1);
				} else {
					map.put(name, count + 1);
				}
			}

			LinkedList<Entry<String, Integer>> list = new LinkedList<>(map.entrySet());

			Collections.sort(list, new Comparator<Entry<String, Integer>>() {
				public int compare(Map.Entry<String, Integer> o1,
				                   Map.Entry<String, Integer> o2) {
					return (o2.getValue()).compareTo(o1.getValue());
				}
			});
			System.out.println("The most common first name is: " + list.get(0).getKey() + " and it occurs: " + list.get(0).getValue() + " times.");
			Instant commonNameEnd = Instant.now();
			long timeElapsedCommonName = Duration.between(commonNameStart, commonNameEnd).toMillis();
			System.out.println("Most common name time: " + timeElapsedCommonName + "ms");

		} catch (FileNotFoundException e) {
			e.printStackTrace();
		}
	}
}

NOTE: There is one thing you need to be aware of if you’re running a plain Java application without the aid of Spring Boot. If you want to use this extra Apache dependency library you need to manually bundle it together with your application’s JAR file into what’s called a fat JAR.

A fat jar (also known as an uber jar) is a self-sufficient archive which contains both classes and dependencies needed to run an application.

Spring Boot "automagically" bundles all of our dependencies together for us, but it’s also got a lot of extra overhead and functionalities that are totally unnecessary for this project, which is why I chose not to use it. It makes the project unnecessarily heavy.

There are plugins available now, but I just wanted a quick and easy way to bundle my one dependency up with my JAR. So I modified the jar task from the Java Gradle plugin. By default, this task produces jars without any dependencies.

I can overwrite this behavior by adding a few lines of code. I just need two things to make it work:

A Main-Class attribute in the manifest file (check, I had three main class files in my demo repo for testing purposes),
And any dependencies jars.

Thanks to Baeldung for the help on making this fat JAR.

jar {
	manifest {
		attributes "Main-Class": "com.example.readFile.readFileJava.ReadFileJavaApplicationLineIterator"
	}

	from {
		configurations.compile.collect {
			it.isDirectory() ? it : zipTree(it)
		}
	}
}

Once the main class file has been defined (and in my demo repo, I made three main class files, which confused my IDE to no end) and the dependencies were included, you can run the ./gradlew assemble command from the terminal and then:

java -cp ./build/libs/readFileJava-0.0.1-SNAPSHOT.jar com.example.readFile.readFileJava.ReadFileJavaApplicationLineIterator

And your program should run with the LineIterator library included.

If you’re using IntelliJ as your IDE, you can also just use its run configurations locally with each of the main files specified as the correct main class, and it should run the three programs as well. See my README.md for more info on this.

Great, now I have three different ways to read and process large text files in Java, my next mission: figure out which way is more performant.

How I evaluated the solution's performance and their results

For performance testing my different Java applications and the functions inside them, I came across two handy, ready-made functions in Java 8: Instant.now() and Duration.between().

What I wanted to do was see if there was any measurable differences between the different ways of reading the same file. So besides the different file read options: FileInputStream, BufferedReader and LineIterator, I tried to keep the code (and timestamps marking the start and stop of each function) as similar as possible. And I think it worked out pretty well.

Instant.now()

Instant.now() does exactly what its name suggests: it holds a single, instantaneous point on the timeline, stored as a long representing epoch-seconds and an int representing a nanosecond-of-seconds. This isn’t incredibly useful all on it’s own, but when it’s combined with Duration.between(), it becomes very useful.

Duration.between()

Duration.between() takes a start interval and an end interval and finds the duration between those two times. That’s it. And that timing can be converted to all sorts of different, readable formats: milliseconds, seconds, minutes, hours, etc.

Here’s an example of implementing Instant.now() and Duration.between() in my files. This one is timing how long it takes to get the line count of the total file.

try {
    LineIterator it = FileUtils.lineIterator(f, "UTF-8");

    // get total line count
    Instant lineCountStart = Instant.now();
    int lines = 0;
    
    System.out.println("Reading file using Line Iterator");

    while (it.hasNext()) {
      String line = it.nextLine();
      lines++;

    }

    System.out.println("Total file line count: " + lines);
    Instant lineCountEnd = Instant.now();
    
    long timeElapsedLineCount =
    Duration.between(lineCountStart, lineCountEnd).toMillis();
    
    System.out.println("Line count time: " +
    timeElapsedLineCount + "ms");
}

Results

Here’s the results after applying Instant.now() and Duration.between() to all of my different file read methods in Java.

I ran all three of my solutions against the 2.55GB file, which contained just over 13 million lines in total.

<figcaption>LineIterator just barely beats out BufferedReader, and both handily beat FileInputStream.</figcaption>

</figure>

As you can see from the table, BufferedReader() and LineIterator() both fared well, but they’re so close in their timings it seems to be up to the developer which they’d rather use.

BufferedReader() is nice because it requires no extra dependencies, but it is slightly more complex to set up in the beginning, with the FileReader() to wrap inside. Whereas LineIterator() is an external library but it makes iterating over the file extra easy after it is included as a dependency.

The percentage improvements are included at the end of the table above as well, for reference.

FileInputStream, in an interesting twist, got blown out of the water by the other two. By buffering the data stream or using the library made specifically for iterating through text files, performance improved by about 73% on all tasks.

Below are the raw screenshots from my terminal for each of my solutions.

Solution #1: FileInputStream()

<figcaption>The solution using FileInputStream() and Scanner().</figcaption>

</figure>

Solution #2: BufferedReader()

<figcaption>The solution using BufferedReader() and FileReader().</figcaption>

</figure>

Solution #3: FileUtils.lineIterator()

<figcaption>The solution using FileUtils.lineIterator().</figcaption>

</figure>

Conclusion

In the end, buffered streams and custom file read libraries are the most efficient ways processing large data sets in Java. At least, for the large text files I was tasked with reading.

Thanks for reading my post on using Java to read really, really large files. If you’d like to see the original posts on Node.js that inspired this one, you can see part 1 and part 2.

Check back in a few weeks, I’ll be writing about Swagger with Express.js or something else related to web development and JavaScript, so subscribe to my newsletter so you don’t miss out.

Thanks for reading, I hope this gives you an idea of how to handle large amounts of data with Java efficiently and performance test your solutions.

References & Further Resources

Github, Read File in Java Repo
Link to the FEC data
Oracle Java Documentation, FileInputStream
Oracle Java Documentation, Scanner
Oracle Java Documentation, BufferedReader
Oracle Java Documentation, FileReader
Apache Commons Java Documentation, LineIterator
Baeldung, Creating a Fat Jar in Gradle
Oracle Java Documentation, Instant
Oracle Java Documentation, Duration

Keep Code Consistent Across Developers The Easy Way — With Prettier & ESLint

Sat, 22 Dec 2018 00:00:00 GMT

<figcaption>A slight exaggeration, but you get the idea.</figcaption>

</figure>

Code formatting: it’s not just for sticklers and grammar nazis

This post is for all the JavaScript developers out there who’ve ever developed any application with another person (or plan to). If that’s you, keep reading. If you’re a solo, rockstar, ninja developer, you can go on about your day.

Now that I’ve got everyone else’s attention, let me give you a hypothetical (real-life) example of what you might encounter upon opening an existing JavaScript codebase that a team of developers has put together.

You may see some (or all) of the following:

Missing semi-colons.
Tons of whitespace between some lines and no whitespace between others.
Run on lines that cause you to scroll right for ages to see everything they contain.
Seemingly random indentation.
Commented out code chunks.
Initialized but unused variables.
Some files that use “strict” JS and others that don’t.
Blocks of code with no spaces or comments anywhere making reading them and deciphering what’s happening that much harder.

Take this piece of code from a codebase I’m currently working in:

<figcaption>This is an example of poorly formatted code. Isn’t that painful to look at? Yeah...</figcaption>

</figure>

Does this sound at all familiar to you? Congrats, you’re on a working dev team, and everyone has their own preferences for how they like to code.

As you might already be able to imagine, what one developer is perfectly fine with (no whitespace anywhere), probably drives another one nuts. And while one dev might indent every line perfectly every time, another dev probably doesn’t care (and knows JavaScript doesn’t care too much either, so s/he can get away with it).

This is a recipe for a hard-to-follow codebase, and I don’t know about you, but I find the act of understanding and writing good, functional code hard enough without having to worry about indentation, spacing, keeping track of variables and simple punctuation fixes.

Not to mention, I hate being that person who makes comments all over a pull request asking for things like that to be cleaned up. Nobody wants to be that person.

Which is what brings me to today’s post.

Today, I’m going to magic away the code inconsistencies of a bunch of different developers with the help of Visual Studio Code, Prettier and ESLint, and leave you with beautiful, uniform code that your development team won’t even have to think about formatting.

Sounds too good to be true? It’s not. I promise.

VS Code & Plugins to the Rescue

Right, so let’s get straight to the solutions to your code formatting problems. In case you’re not familiar with it, VS Code is the de facto JavaScript IDE today.

And for good reason: it’s free, it’s easy to get up and running, and it’s just an absolute pleasure to work with. That’s just the tip of the iceberg, but there’s a million other articles expounding on the awesomeness of VS Code, so I’ll leave that to them.

Before this, I was a supporter of all things JetBrains like WebStorm and IntelliJ, but after giving Visual Studio Code a try, I’m sold.

Which leads to...

Step 1: If you haven’t downloaded it yet, download VS Code now. It’s a critical piece of our puzzle to solve this formatting and code consistency problem.

Step 2: Getting started with plugins. One of my favorite things about VS Code is the extremely robust plugin ecosystem it has, and the ease of integrating said plugins. You set them up once, then you can forget about them, and they just keep working, project after project.

There’s two plugins you need in your arsenal today: Prettier and ESLint.

If you open up your Extensions tab on the side of your VS Code window, you’ll see a host of extensions made for VS Code.

<figcaption>See that square thing on the far left? Click that to open up the plugin search for VS Code.</figcaption>

</figure>

But before I have you download these plugins, let me give you a quick overview of what these extensions can do for you.

Prettier

</figure>

Very simply, Prettier is a "VS Code package to format your JavaScript / TypeScript / CSS" - Prettier site

In practice, Prettier is an opinionated code formatter that does all sorts of useful things like:

Changes all single quotes to double quotes.
Adds missing semicolons.
Puts spaces between curly braces or brackets and variables.
Sets standard tab width.

This is only a small fraction of what Prettier cares about, and in VS Code it’s really easy to override any rules that you’re not a fan of, which I’ll get to in a bit.

Prettier is made for keeping code formatting consistent, and the VS Code plugin, it works based on a .prettierrc file in the root of a project. It will keep your code clean and easy to read, and the same across all developers on the team.

One of the biggest selling points for me?

Prettier can be set up in VS Code to auto-format code on every save.

I don’t have to think about it, I don’t have to set up complicated file watcher tasks, I change one setting in VS Code and it just works. Every time. Period. The end.

I’ll show you how to do this too, but before I get there, I need you to download Prettier for VS Code.

Open up the Extensions Marketplace in your VS Code window (it’s the little square looking doodad under the debugger).

This is the first plugin you’ll be searching for in VS Code’s extensions. If you type "Prettier" into the search bar, your results should look something like this:

<figcaption>Go for Prettier - Code Formatter. This is the one you want.</figcaption>

</figure>

You see the first result? The one with 5.7 million downloads and counting? That’s the one you want to install. When in doubt, look for the extension with the most downloads that’s closest to what you think the name of the extension is, and go with it.

If you think the "Install" buttons on VS Code look like labels or banners instead of finished, clickable buttons, I agree with you. But that’s a debate for another day.

Once the Prettier extension is installed, we’re ready to move on to our second plugin: ESLint.

ESLint

</figure>

The thinking behind ESLint is: “JavaScript, being a dynamic and loosely-typed language, is especially prone to developer error. Without the benefit of a compilation process, JavaScript code is typically executed in order to find syntax or other errors. Linting tools like ESLint allow developers to discover problems with their JavaScript code without executing it.” — ESLint site

ESLint is great for the more specific, less generic code styles that you want your development team to adhere to. Unless you specifically set it up, ESLint won’t autofix or rewrite your code, but it will let you know in a straightforward way that there’s “rules” being broken.

Whether those rules include commented out code, unused variables or missing prop-types is up to you and your team, but with an .eslintrc file in place in your project, it sets up guardrails around the project to ensure each JavaScript file written adheres to the same standards as the others, regardless of who’s working on it.

The ESLint plugin for VS Code is an extension of the official, open source ESLint utility used by the likes of Google, Facebook, Netflix and more.

If ESLint is good enough for them, it’s certainly good enough for me.

Here’s what you’ll see when searching for ESLint in the VS Code extension marketplace:

<figcaption>The one with 15 million downloads - that’s the one you want.</figcaption>

</figure>

One of the biggest benefits to ESLint is that each “rule” set in ESLint is entirely standalone. Every one can be turned on or off, new rules can be added, unnecessary rules can be ignored, and there’s a ton of rules that are already documented to be included with just a line of code.

No rules are enabled by default, so you’re free to add them as you see fit, but a lot of development teams like to start with something like Airbnb’s ESLint file, which is packaged as a handy NPM module(npm i eslint-config-airbnb), and adjust as they need to suit their needs.

As an aside: Airbnb’s whole style guide is a great example of how to approach writing JavaScript, in general. I figure, 79,000+ stars on Github is a pretty good indicator of quality. But again, that’s a tangent for another post.

For now, let’s download Prettier and ESLint, and move on to getting them running in VS Code.

Setting up and configuring Prettier and ESLint

This is the fun part. It’s the super, super easy part, too. You’ll be amazed just how painless it is, and it just makes me even happier that VS Code exists.

Once you’ve downloaded these plugins, it’s time to enable them.

To check that your new plugins are installed and enabled, while still in the extensions screen, you can click the three dots at the top right of the search input and click the "Show Installed Extensions or "Show Enabled Extensions" selection from the dropdown, and if your new extensions are downloaded/enabled ESLint and Prettier will show up there.

<figcaption>Both ESLint and Prettier are installed and enabled for my VS Code workspaces.</figcaption>

</figure>

If either is not enabled, just click the extension and click the "Enable" button (yep, it looks like a label, but it’s a button). Then restart your VS Code program for the changes to take effect.

<figcaption>If ESLint was not enabled, the Disable button would say "Enable" and I could click it to turn it on from there.</figcaption>

</figure>

Default user settings

Great, now both plugins should be enabled and we’re ready to get started. As I said before, Prettier has lots of settings enabled by default, and you can see exactly what’s enabled / disabled and make any changes you’d like to take effect globally, here.

To get to these settings, go to Code > Preferences > Settings and then search "prettier" in the input box. Once you’re in general Settings, click the curly braces in the top right corner of the window and it will take you to the "Default User Settings", where you can see all the Prettier settings and what they default to.

<figcaption>Default Prettier configuration settings in my VS Code.</figcaption>

</figure>

From here, you can do the same thing by putting "eslint" in the search box, and see all the rules in place by default for that extension.

<figcaption>Default ESLint configuration settings in my VS Code.</figcaption>

</figure>

And that’s the defaults. Good, now let’s customize it a little bit, because I want to make my development life even easier.

Custom user settings

VS Code offers this really handy way to override any default settings of specific extensions with what it calls "User Settings".

All this is, is a file that VS Code generates with settings that are specific to your text editor. If you want to override a specific rule, just hover your cursor over the rule to modify, then click the pencil that appears at the left, and a new window titled "User Settings" will open up and add that rule as a modifiable setting there.

Here’s an example of what my user settings look like on the right in the screenshot below. I’ve overwritten a couple of the Prettier settings: singleQuote and trailingComma, and I’ve enabled ESLint’s autoFixOnSave() function.

By far, the best thing I enabled, though, is Prettier’s formatOnSave(), so I don’t even have to think about it. This was done through the following code snippet.

"[javascript]": {
    "editor.formatOnSave": true
}

<figcaption>Click the Edit pencil on the left, and the rule gets added to your custom "User Settings" on the right.</figcaption>

</figure>

Prettier’s formatOnSave() is on a per-language basis, so if you want to add in formatting for CSS or HTML or some other language, you just need to add it here. And then it’s ready to go.

Make a change in a JavaScript file, save the change, and the code reformats itself according to Prettier. It’s glorious. Truly, one of my favorite features, by far.

Project-specific config files

And finally, the project specific configuration files.

I find the default settings straight through the VS Code extension to be enough for me when I’m coding my own projects, but for a dev team, to be absolutely sure everyone’s coding to the same standards, that file could be included to take out all the guesswork.

Anyway, for ESLint to work best, it needs an .eslintrc file at the root of the project. Here’s an example of what that file could look like.

<figcaption>Example of an .eslintrc file.</figcaption>

</figure>

This is a pared down one, but you get the idea. Any rules to be linted against are added at the bottom of the file.

And once this file is present, if there’s a rule that’s being broken, it will appear in the "Problems" section where the built-in terminal runs in the VS Code window.

<figcaption>ESLint doing its linting.</figcaption>

</figure>

Without a project specific .eslintrc file, unless you have a lot of hard and fast rules you want to set in your custom User Settings, you won’t get a lot of assistance from ESLint. Personally, I like the project specific ESLint file, it makes it crystal clear what the rules and standards are, and unlike Prettier, which is very opinionated by default, ESLint won’t complain unless you tell it to.

And here's a basic .prettierrc file.

<figcaption>Example of a .prettierrc file.</figcaption>

</figure>

To wrap it up, the configuration settings in VS Code are a little like CSS and its specificity rules— the more specific the settings file, the higher the priority it is given by VS Code. If there’s no project specific config file, VS Code falls back to the custom user settings, and if there’s no custom user settings it looks for default settings last of all.

Conclusion

There you have it. Your dev team is now set up for success — at least when it comes to consistent, well-formatted, and clean JavaScript code. VS Code makes it so simple to implement thorough code linting and opinionated code formatting with little effort on the developer’s part. There’s no reason not to get on board with that.

These simple tips will speed up development and keep everyone’s code looking consistent, even with all of their unique coding styles.

Check back in a few weeks, I’ll be writing reading really large files in Java (a follow up to my two part series on reading really large files with Node.js) or something else related to web development, so please follow me so you don’t miss out.

Thanks for reading, I hope this gives you an idea of how to easily keep your code consistent with Prettier and ESLint.

References & Further Resources

Getting Started with Puppeteer & Headless Chrome for End-to-End Testing

Sat, 08 Dec 2018 00:00:00 GMT

</figure>

Inspiration for this post

This blog post was inspired by a talk I saw at the Connect.Tech Conference in Atlanta this year, entitled: "Using Puppeteer to Write End-to-End Tests That Run On The Browser."

The speaker, Taylor Krusen, gave a great talk on Puppeteer and headless Chrome and using them together to take end-to-end testing to the next level. I’d looked briefly into Puppeteer in the past, but since I couldn’t find any great examples of how to get started with it, I’d passed and chosen to use Cypress.io for my testing needs instead.

After the talk though, I was inspired to give Puppeteer another shot and I went a step further, figuring out how to run Puppeteer and how to use it with Jest, the testing framework most commonly associated with React.

And I want to share that today to hopefully help others get started with their own application testing suites.

What is end-to-end testing & why is it important?

<figcaption>End to end tests make up the smallest part of the testing pyramid, as defined by Google, but they are still a crucial part of it.</figcaption>

</figure>

Before I dive into how to use Puppeteer for end-to-end testing, I think it’s important to briefly explain what it is and why it’s important for web development today.

Techopedia gives a good definition of end-to-end testing:

End-to-end testing is a methodology used to test whether the flow of an application is performing as designed from start to finish. The purpose of carrying out end-to-end tests is to identify system dependencies and to ensure that the right information is passed between various system components and systems.

Essentially, end-to-end testing replicates, in an automated fashion, exactly what a user could do while interacting with the application or browser. And this is done to ensure the application works properly and as expected.

As web applications get more and more complex and full featured, it becomes harder and harder for manual QA teams to keep up with testing every single piece of functionality each time the application is ready to deploy a new feature. End-to-end testing frameworks were created to take the burden (and repetitive task) of checking functionality out of the hands of people and have machines (which can move much faster than actual users and exactly replicate a test every time) shoulder more of the load.

Ultimately, it still comes down to us, as developers, to ensure our applications work the way we expect, but testing helps us to better ensure we’re not introducing new bugs into our code or compromising existing functionality, which is why it’s so important.

What are Headless Chrome & Puppeteer?

</figure>

The next thing to cover is what exactly are Headless Chrome and Puppeteer, and why are they always bundled together?

Puppeteer’s GitHub and npm pages do a great job of explaining exactly what Puppeteer is:

Puppeteer is a Node library which provides a high-level API to control Chrome or Chromium over the DevTools Protocol.

It goes on to say Puppeteer runs headless Chrome or Chromium instances by default, which is why they’re always mentioned in tandem.

Most things that you can do manually in the browser can be done using Puppeteer!

Puppeteer was made for Chrome by the Google Chrome DevTools team to help further automated/headless browser testing and make it less painful for the rest of us.

The biggest drawback is that Puppeteer only works with Chrome. I have had the good fortune of working on applications where our users use Chrome exclusively, which made Puppeteer a very good option for end-to-end testing, but if the user base is more varied in its browser preferences, it may be wise to look into other testing frameworks like Cypress.io, TestCafe, or Selenium Web Driver.

But today, at least, I get to talk about Puppeteer.

Puppeteer examples

Now it’s time to get down to the business of getting Puppeteer up and running.

Getting started with Puppeteer is actually incredibly easy — just type npm i puppeteer and you’re ready to go. Yes, it really is that simple. It’s this easy because the Chrome DevTools team conveniently packages each version of Puppeteer with a version of Chromium (the headless Chrome part) that it’s guaranteed to run with.

NOTE: Puppeteer requires at least Node v6.4.0, but the examples below use async/await which is only supported in Node v7.6.0 or greater. Just so ya know.

Let’s take a look at what Puppeteer can automate.

If you’d like to play with these examples yourself, you can download my GitHub repo here.

You can also click on the file name for each of these code samples below to see the actual code in GitHub.

Take a screenshot of a page: saveFileExample.js

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://www.google.com');
  await page.screenshot({ path: 'google.png' });

  await browser.close();
})();

As you’ll see with all these scripts, as long as puppeteer is imported at the top of the file, it’s good to go and all the methods packaged along with Puppeteer like newPage() and browser are instantly available to you.

This first sample script goes to the Google home page and takes a photo of the page and saves it to a .png file inside of the repo named "google.png".

The code is pretty self-explanatory. To begin, a headless Chrome browser is launched, a new page is opened, the URL for that page is set to Google, a screenshot is taken for posterity and the browser is closed. The whole thing is clean, elegant, asynchronous and wrapped within an IIFE (immediately invoked function expression) so as soon as the script is called, the function runs.

Create a PDF: createPDF.js

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://www.thinkgeek.com/', { waitUntil: 'networkidle2' });
  await page.pdf({ path: 'thinkGeek.pdf', format: 'A4' });

  await browser.close();
})();

The second Puppeteer example goes to a website (the fun retail website, ThinkGeek, in this case) and makes the site page it visits into a PDF.

Similar to the first script, this one sets up Puppeteer, creates a new browser page, goes to ThinkGeek’s website and then uses the page.pdf() method to make a PDF of the whole page. That PDF is once again, saved into the project folder under the name "thinkGeek.pdf". Pretty simple.

Google something: searchGoogle.js

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://www.google.com');

  const searchBox = await page.$('input[type=text]');
  await searchBox.type('cookies');
  const inputElement = await page.$(
    'input[type=submit][value="Google Search"]',
  );
  await inputElement.click();
  const [response] = await Promise.all([
    page.waitForNavigation(),
    page.once('load', () => console.log('Cookies loaded!')),
  ]);

  await page.screenshot({ path: 'cookies.png' });
  await browser.close();
})();

This third example is probably something more like what would be tested in an end-to-end test — it’s automating a Google search (and taking a screenshot to confirm the search worked, in headless mode).

In this script, Puppeteer launches a new browser page, it goes to Google, it waits for the search box and the search button to be visible and types "cookies" into the input, then it waits for the page to navigate to the cookies search results and takes a screenshot before closing the browser at the end.

And that’s all there is to it. To see more of what Puppeteer can do, I recommend checking out their very thorough documentation — there’s a whole lot of info and functionality to discover there.

Debugging Puppeteer

Now that Puppeteer is running, I want to call out some awesome tools that are built into it to make debugging it much less painful. Here's the code snippet you need, which I'll break down next.

const browser = await puppeteer.launch({
  headless: false,
  slowMo: 500,
  devtools: true
})

Turn off headless mode

The first (and most obvious one) is how to turn off Chrome’s headless mode; meaning you can watch the automated browser open and attempt to go through the steps laid out for it.

This is a cinch. Inside the script where Puppeteer is launched (pretty much the first thing that happens when the script runs), just add the following line: puppeteer.launch({headless: false});.

Enable SlowMo

SlowMo is just what it sounds like, it slows down Puppeteer’s operations so normal humans can see what’s going on. It can be instantiated in the same command where headless is set to false for Puppeteer; the puppeteer.launch() command. Just add puppeteer.launch({slowMo: 500}) or however many milliseconds you want to slow down the actions by.

Auto-open DevTools panel

The third debugging trick I have is how to tell Puppeteer whether to auto-open a Chrome DevTools panel with each tab. So if you need to, you can catch things being logged to the console inside the automated test run.

Once again, inside the initial command launching Puppeteer, add in puppeteer.launch({headless: false, devtools: true}); to open a DevTools panel.

NOTE: If this option is true, the headless option will be set false.

You can even console.log() events from the headless Chrome browser to your own terminal by simply throwing this method from page.exposeFunction() into the mix. Right after the page object is defined in the Puppeteer script, you can do:

page.on("console", msg => console.log(`Page Console: ${msg.text()}`));

to catch messages from console.log.

That should help get you on your way to debugging your Puppeteer headless Chrome instances and tests like a pro.

More cool features of Puppeteer

In addition to debugging Puppeteer, I wanted to point out a few more cool features that it offers. If you’re interested in learning more about any of them, I’d recommend checking out their npm page for more info.

Puppeteer can:

Run in a Docker container or serverless environment,
Intercept network requests,
Capture performance info,
Test a Chrome extension,
Run code in a page, and
Emulate Chrome on a mobile device.

That’s a lot of additional useful things it can be used for. The team really wants to make it useful to devs in many ways.

Bonus: React & Jest & Puppeteer

</figure>

I’d be remiss if I didn’t also briefly mention how to integrate Puppeteer into a React project with Jest tests. React is my current JavaScript framework of choice (and it seems the same for many JS developers), so I created a quick setup and test script using the two together, also housed in the my sample GitHub repo here.

Configuration for Jest & Puppeteer

Jest has some very helpful documentation to smoothly work Puppeteer into its own testing framework, and there’s even an entire npm package named jest-puppeteer to help the process along. Here’s what to install and configure to make Jest and Puppeteer play nice.

npm install --save-dev jest-puppeteer puppeteer jest

Once everything’s been installed, you can either configure your Jest config file, Webpack file or your package.json file to let Jest know that Puppeteer’s there and ready to be used. Here’s how I did it in the package.json.

Right above the scripts in the file, I added this snippet, and inside the scripts, I included a test command which just runs Jest.

package.json

"jest": {
    "preset": "jest-puppeteer"
  },
  "scripts": {
    "test": "jest",
    /* more scripts here */
  }

Now, I’m ready to write tests.

Jest & Puppeteer Test: google.test.js

describe('Google', () => {
  beforeAll(async () => {
    await page.goto('https://google.com');
  });

  it('should display "google" text on page', async () => {
    await expect(page).toMatch('google');
  });
});

If you’ve ever written a Jest test before, the syntax should look familiar. Since I don’t have a full React project as part of my sample repo, I’ll use this simple example test where Jest should go to the Google homepage and find the text "Google” on said page after it’s loaded.

From my repo, I set up the npm script to run the jest command when npm run test is entered into the terminal. Jest looks for any files with the *.test.js syntax, by default and tries to run those. When it finds my test file entitled google.test.js it runs it (headlessly, unless I were to specify differently) and logs out the test results to the console, like so.

If tests failed, you can debug just as you normally would (and even turn on the automated Chrome browser if you need to see what’s messing up), and if not, your computer screen can be test free as your end-to-end tests still run in the background.

Awesome. That doesn’t seem so hard, now does it?

Conclusion

Puppeteer is a powerful tool developed by the Google DevTools team to make headless, automated browser testing easier in every way. There’s less configuration to get started, the syntax makes more sense, and it has uses that extend far beyond just testing web apps during development. Likewise, it’s easy to integrate into React applications to automate Jest tests to run headlessly as well.

Check back in a few weeks, I’ll be writing about something else related to web development, so subscribe to my free newsletter so you don’t miss out.

Thanks for reading, I hope this gives you an idea of how to get started using Puppeteer for your own end-to-end testing and integrating with React and Jest tests.

References & Further Resources

Puppeteer Jest sample repo on GitHub
End-to-end testing on Techopedia
Puppeteer docs on npm
Puppeteer repo on GitHub
Puppeteer API docs on GitHub
Jest with Puppeteer site docs

Password Reset Emails In Your React App Made Easy with Nodemailer

Thu, 22 Nov 2018 00:00:00 GMT

</figure>

Password Resets in a MERN Application

Before I actually attempted to build in an email-based password reset for my MERN app, I thought it would be tougher to do. From everything I’d heard before, sending emails in a JavaScript application was painful, but I still wanted to attempt it.

For a few months, to hone my full stack JavaScript skills, I’ve been slowly building and adding on to a user registration service.

First, I built it with a React frontend, an Express / Node.js backend and a Docker-powered MySQL database. I used a docker-compose.yml to start the whole app with one command (if you’d like to read more about my use of Docker for development, you can see this blog post).

After I’d gotten that working, I added in authorization to the app using Passport.js and JSON Web Tokens (JWTs). You can read about the fun (pain) of that here, if you’re curious. And it took me a while — I ran into a bunch of road blocks and obstacles that stalled me out multiple times. But grit and my inability to let go of a problem once it’s taken root in my brain prevailed, and I figured it out and moved on.

When I decided to tackle sending password reset links via email (just like real sites when users, myself included, inevitably forget their password), I figured I was in for more pain. It just couldn’t be too easy, no matter that practically every site out there has this very functionality. But I was wrong. And I’m so glad I was.

Today, I'll show you how to set up your own password reset system with help from Nodemailer.

Nodemailer: the magic bullet

</figure>

Once I started googling around looking for solutions to my password reset feature, I came across a number of articles recommending Nodemailer.

When I visited the website, the first lines I read were:

Nodemailer is a module for Node.js applications to allow easy as cake email sending. The project got started back in 2010 when there was no sane option to send email messages, today it is the solution most Node.js users turn to by default. — Nodemailer

And you know what? It wasn’t kidding. Easy as cake isn’t too far wrong.

Of course, before I got started, I did a little more digging to make sure I was placing my faith in a decent piece of technology, and what I saw on npm and GitHub put my mind at ease.

Nodemailer has:

Over 615,000 weekly downloads from NPM,
Over 10,000 stars on Github,
206 releases to date,
Over 2,500 dependent packages,
And it’s been around in some form or fashion since 2010.

Ok, that seemed good enough for me to give it a shot in my own project.

Implementation of Nodemailer in my code (frontend and backend)

I didn’t need anything fancy for my password reset, just:

A way to send an email to a user’s address.
That email would contain a link that would redirect them to a protected page on my site where they could reset their password.
Then they could log in using their new password.
I also wanted the password reset link to expire after a certain time period for better security.

Here’s how I did it.

Frontend code (client folder): send reset email

I’ll start with the React code first, because I had to have a page where users could enter their email addresses and fire off the email with the reset link.

ForgotPassword.js

/* eslint-disable no-console */
import React, { Component } from 'react';
import TextField from '@material-ui/core/TextField';
import axios from 'axios';
import {
  LinkButtons,
  SubmitButtons,
  registerButton,
  homeButton,
  forgotButton,
  inputStyle,
  HeaderBar,
} from '../components';

const title = {
  pageTitle: 'Forgot Password Screen',
};

class ForgotPassword extends Component {
  constructor() {
    super();

    this.state = {
      email: '',
      showError: false,
      messageFromServer: '',
      showNullError: false,
    };
  }

  handleChange = name => (event) => {
    this.setState({
      [name]: event.target.value,
    });
  };

  sendEmail = async (e) => {
    e.preventDefault();
    const { email } = this.state;
    if (email === '') {
      this.setState({
        showError: false,
        messageFromServer: '',
        showNullError: true,
      });
    } else {
      try {
        const response = await axios.post(
          'http://localhost:3003/forgotPassword',
          {
            email,
          },
        );
        if (response.data === 'recovery email sent') {
          this.setState({
            showError: false,
            messageFromServer: 'recovery email sent',
            showNullError: false,
          });
        }
      } catch (error) {
        console.error(error.response.data);
        if (error.response.data === 'email not in db') {
          this.setState({
            showError: true,
            messageFromServer: '',
            showNullError: false,
          });
        }
      }
    }
  };

  render() {
    const { email, messageFromServer, showNullError, showError } = this.state;

    return (
      <div>
        <HeaderBar title={title} />
        <form className="profile-form" onSubmit={this.sendEmail}>
          <TextField
            style={inputStyle}
            id="email"
            label="email"
            value={email}
            onChange={this.handleChange('email')}
            placeholder="Email Address"
          />
          <SubmitButtons
            buttonStyle={forgotButton}
            buttonText="Send Password Reset Email"
          />
        </form>
        {showNullError && (
          <div>
            <p>The email address cannot be null.</p>
          </div>
        )}
        {showError && (
          <div>
            <p>
              That email address isn't recognized. Please try again or
              register for a new account.
            </p>
            <LinkButtons
              buttonText="Register"
              buttonStyle={registerButton}
              link="/register"
            />
          </div>
        )}
        {messageFromServer === 'recovery email sent' && (
          <div>
            <h3>Password Reset Email Successfully Sent!</h3>
          </div>
        )}
        <LinkButtons buttonText="Go Home" buttonStyle={homeButton} link="/" />
      </div>
    );
  }
}

export default ForgotPassword;

Ok, I know this is a big code snippet but I’ll break it down.

If you want to copy/paste the actual code, you can see the whole repo here.

What you should really focus on here is the sendEmail() function and the render() method of the component. The rest is just setting initial state and variables, and styling of buttons and elements.

The render() method

render() {
    const { email, messageFromServer, showNullError, showError } = this.state;

    return (
      <div>
        <HeaderBar title={title} />
        <form className="profile-form" onSubmit={this.sendEmail}>
          <TextField
            style={inputStyle}
            id="email"
            label="email"
            value={email}
            onChange={this.handleChange('email')}
            placeholder="Email Address"
          />
          <SubmitButtons
            buttonStyle={forgotButton}
            buttonText="Send Password Reset Email"
          />
        </form>
        {showNullError && (
          <div>
            <p>The email address cannot be null.</p>
          </div>
        )}
        {showError && (
          <div>
            <p>
              That email address isn&apos;t recognized. Please try again or
              register for a new account.
            </p>
            <LinkButtons
              buttonText="Register"
              buttonStyle={registerButton}
              link="/register"
            />
          </div>
        )}
        {messageFromServer === 'recovery email sent' && (
          <div>
            <h3>Password Reset Email Successfully Sent!</h3>
          </div>
        )}
        <LinkButtons buttonText="Go Home" buttonStyle={homeButton} link="/" />
      </div>
    );
  }

Notice inside the render() method, that I have a simple input box for the user to enter his/her email address, and a submit button which triggers the this.sendEmail() function when clicked. Beyond that, I have a little error and success handling built in based on if the user hasn’t entered an email, if the server responds back the email was successfully sent or it wasn’t a recognized address.

The sendEmail() function

  sendEmail = async (e) => {
    e.preventDefault();
    const { email } = this.state;
    if (email === '') {
      this.setState({
        showError: false,
        messageFromServer: '',
        showNullError: true,
      });
    } else {
      try {
        const response = await axios.post(
          'http://localhost:3003/forgotPassword',
          {
            email,
          },
        );
        if (response.data === 'recovery email sent') {
          this.setState({
            showError: false,
            messageFromServer: 'recovery email sent',
            showNullError: false,
          });
        }
      } catch (error) {
        console.error(error.response.data);
        if (error.response.data === 'email not in db') {
          this.setState({
            showError: true,
            messageFromServer: '',
            showNullError: false,
          });
        }
      }
    }
  }

For all of my HTTP requests, I’m using the Axios library, which just makes it really easy to make AJAX calls to the server, even easier than the built in fetch() web API, in my opinion.

When the user enters his/her email, I make a POST request to the server, and wait for a response. If the email address isn’t found, I can tell the user they entered it wrong or if they’re new, they can go to the register page and create an account, and if the address does match one in my database, they’ll get back a success message saying the password reset link has been sent to their email address.

Let’s move on to the backend code now.

Backend code (API Folder): send reset email

forgotPassword.js

/* eslint-disable max-len */
/* eslint-disable no-console */
import crypto from 'crypto';
import User from '../sequelize';

require('dotenv').config();

const nodemailer = require('nodemailer');

module.exports = (app) => {
  app.post('/forgotPassword', (req, res) => {
    if (req.body.email === '') {
      res.status(400).send('email required');
    }
    console.error(req.body.email);
    User.findOne({
      where: {
        email: req.body.email,
      },
    }).then((user) => {
      if (user === null) {
        console.error('email not in database');
        res.status(403).send('email not in db');
      } else {
        const token = crypto.randomBytes(20).toString('hex');
        user.update({
          resetPasswordToken: token,
          resetPasswordExpires: Date.now() + 3600000,
        });

        const transporter = nodemailer.createTransport({
          service: 'gmail',
          auth: {
            user: `${process.env.EMAIL_ADDRESS}`,
            pass: `${process.env.EMAIL_PASSWORD}`,
          },
        });

        const mailOptions = {
          from: 'mySqlDemoEmail@gmail.com',
          to: `${user.email}`,
          subject: 'Link To Reset Password',
          text:
            'You are receiving this because you (or someone else) have requested the reset of the password for your account.\n\n'
            + 'Please click on the following link, or paste this into your browser to complete the process within one hour of receiving it:\n\n'
            + `http://localhost:3031/reset/${token}\n\n`
            + 'If you did not request this, please ignore this email and your password will remain unchanged.\n',
        };

        console.log('sending mail');

        transporter.sendMail(mailOptions, (err, response) => {
          if (err) {
            console.error('there was an error: ', err);
          } else {
            res.status(200).json('recovery email sent');
          }
        });
      }
    });
  });
};

The backend code is a little more involved. This is where Nodemailer comes into play.

When the user hits the forgotPassword route on the backend with the email address they entered, the first thing Sequelize (my SQL ORM) does is check if that email exists in my database. If it doesn’t the user gets notified they may have mistyped it, if it does exist, then a series of other events starts.

None of the following steps is very difficult, it’s just chaining them all together that was a little tricky, at first.

1. Generate an expiring token for the user

 const token = crypto.randomBytes(20).toString('hex');
        user.update({
          resetPasswordToken: token,
          resetPasswordExpires: Date.now() + 3600000,
        });

The first step after confirming the email belongs to a user in the database, is generating a token that can be attached to the user’s account and setting a time limit for that token to be valid.

Node.js has a built in module called Crypto, which provides cryptographic functionality, which is a fancy way of saying, I can generate a unique hash token easily using the JavaScript crypto.randomBytes(20).toString('hex');. Then, I save that new token to my user’s profile in the database under the column name resetPasswordToken. I also set a timestamp for how long that token will be valid. I made mine valid for one hour after sending the link: Date.now() + 3600000.

2. Create Nodemailer transport

 const transporter = nodemailer.createTransport({
          service: 'gmail',
          auth: {
            user: `${process.env.EMAIL_ADDRESS}`,
            pass: `${process.env.EMAIL_PASSWORD}`,
          },
        });

Next, I created the transporter which is actually the account sending the password reset email link.

I chose to use Gmail, because I use Gmail personally, and I created a new dummy account to send the emails. Since I don’t want to give out the credentials for that account to anyone, I put the credentials into a .env file that is included in my .gitignore so it never gets committed to GitHub or anywhere else.

The npm package dotenv is used to read the contents of the file and insert the email address and password for Nodemailer’s [createTransport()] function to pick up.

3. Create mail options

 const mailOptions = {
          from: 'mySqlDemoEmail@gmail.com',
          to: `${user.email}`,
          subject: 'Link To Reset Password',
          text:
            'You are receiving this because you (or someone else) have requested the reset of the password for your account.\n\n'
            + 'Please click on the following link, or paste this into your browser to complete the process within one hour of receiving it:\n\n'
            + `http://localhost:3031/reset/${token}\n\n`
            + 'If you did not request this, please ignore this email and your password will remain unchanged.\n',
        };

The third step is creating the email template or mailOptions as Nodemailer calls it that the user will see (and this is also where the verified email address they pass from the frontend input gets used).

There are whole third-party libraries for making great looking emails to go with the Nodemailer module, but I just wanted a bare bones email, so I made this one myself.

It contains the from email address (mySqlDemoEmail[at]gmail.com, for me), the user’s email goes in the to box, the subject line is something along the lines of reset password link, and the text is a simple string containing a little info and the website’s URL reset route including the token I created earlier, tacked on to the end. This will allow me to verify the user is who they say they are when they click the link and go to the site to reset their password.

4. Send Email

        transporter.sendMail(mailOptions, (err, response) => {
          if (err) {
            console.error('there was an error: ', err);
          } else {
            res.status(200).json('recovery email sent');
          }
        });

The final step of this file actually putting together the pieces I created: the transporter, the mailOptions, the token and using Nodemailer’s sendMail() function. If it works, I’ll get back a 200 response, which I then use to trigger a success call to the client, and if it fails, I log out the error to see what went wrong.

Enabling Gmail to send reset emails

There’s an extra gotcha to be aware of when setting up the transporter email that all the emails are sent from, at least, when using Gmail.

In order to be able to send emails from an account, 2-Step verification must be disabled, and a setting titled ‘Allow less secure apps’ must be toggled to on. See screenshot below. To do this, I went to my settings here, and turned it on.

Now, I could send reset emails with no problems. If you’re having trouble, check Nodemailer’s FAQs for more help.

<figcaption>This is the screen you should see where you can turn ‘on’ less secure apps. Just one more reason to use some dummy email account instead of your actual Gmail account too. </figcaption>

</figure>

Frontend code: update password screen

Great, now users should be getting reset emails in their inbox that look something like this.

<figcaption>It’s a simple email, but it does what I need it to do.</figcaption>

</figure>

If you notice, the third line is a link to my website (running locally on port 3031), to another page called ‘Reset’, followed by the hashed token I generated back in step 1 with the Node.js crypto module.

When a user clicks this link, they’re directed to a new page in the application entitled ‘Password Reset Screen’, which can only be accessed with a valid token. If the token has expired or is otherwise invalid, the user will see an error screen with links to go home or attempt to send a new password reset email.

Here’s the React code for the reset screen.

ResetPassword.js

/* eslint-disable no-console */
import React, { Component } from 'react';
import PropTypes from 'prop-types';
import axios from 'axios';
import TextField from '@material-ui/core/TextField';

import {
  LinkButtons,
  updateButton,
  homeButton,
  loginButton,
  HeaderBar,
  forgotButton,
  inputStyle,
  SubmitButtons,
} from '../components';

const loading = {
  margin: '1em',
  fontSize: '24px',
};

const title = {
  pageTitle: 'Password Reset Screen',
};

export default class ResetPassword extends Component {
  constructor() {
    super();

    this.state = {
      username: '',
      password: '',
      updated: false,
      isLoading: true,
      error: false,
    };
  }

  async componentDidMount() {
    const {
      match: {
        params: { token },
      },
    } = this.props;
    try {
      const response = await axios.get('http://localhost:3003/reset', {
        params: {
          resetPasswordToken: token,
        },
      });
      if (response.data.message === 'password reset link a-ok') {
        this.setState({
          username: response.data.username,
          updated: false,
          isLoading: false,
          error: false,
        });
      }
    } catch (error) {
      console.log(error.response.data);
      this.setState({
        updated: false,
        isLoading: false,
        error: true,
      });
    }
  }

  handleChange = name => (event) => {
    this.setState({
      [name]: event.target.value,
    });
  };

  updatePassword = async (e) => {
    e.preventDefault();
    const { username, password } = this.state;
    const {
      match: {
        params: { token },
      },
    } = this.props;
    try {
      const response = await axios.put(
        'http://localhost:3003/updatePasswordViaEmail',
        {
          username,
          password,
          resetPasswordToken: token,
        },
      );
      if (response.data.message === 'password updated') {
        this.setState({
          updated: true,
          error: false,
        });
      } else {
        this.setState({
          updated: false,
          error: true,
        });
      }
    } catch (error) {
      console.log(error.response.data);
    }
  };

  render() {
    const {
 password, error, isLoading, updated 
} = this.state;

    if (error) {
      return (
        <div>
          <HeaderBar title={title} />
          <div style={loading}>
            <h4>Problem resetting password. Please send another reset link.</h4>
            <LinkButtons
              buttonText="Go Home"
              buttonStyle={homeButton}
              link="/"
            />
            <LinkButtons
              buttonStyle={forgotButton}
              buttonText="Forgot Password?"
              link="/forgotPassword"
            />
          </div>
        </div>
      );
    }

    if (isLoading) {
      return (
        <div>
          <HeaderBar title={title} />
          <div style={loading}>Loading User Data...</div>
        </div>
      );
    }

    return (
      <div>
        <HeaderBar title={title} />
        <form className="password-form" onSubmit={this.updatePassword}>
          <TextField
            style={inputStyle}
            id="password"
            label="password"
            onChange={this.handleChange('password')}
            value={password}
            type="password"
          />
          <SubmitButtons
            buttonStyle={updateButton}
            buttonText="Update Password"
          />
        </form>

        {updated && (
          <div>
            <p>
              Your password has been successfully reset, please try logging in
              again.
            </p>
            <LinkButtons
              buttonStyle={loginButton}
              buttonText="Login"
              link="/login"
            />
          </div>
        )}
        <LinkButtons buttonText="Go Home" buttonStyle={homeButton} link="/" />
      </div>
    );
  }
}

ResetPassword.propTypes = {
  // eslint-disable-next-line react/require-default-props
  match: PropTypes.shape({
    params: PropTypes.shape({
      token: PropTypes.string.isRequired,
    }),
  }),
};

And here’s the three main pieces of this component that do the heavy lifting.

The initial componentDidMount() lifecycle method

  async componentDidMount() {
    const {
      match: {
        params: { token },
      },
    } = this.props;
    try {
      const response = await axios.get('http://localhost:3003/reset', {
        params: {
          resetPasswordToken: token,
        },
      });
      if (response.data.message === 'password reset link a-ok') {
        this.setState({
          username: response.data.username,
          updated: false,
          isLoading: false,
          error: false,
        });
      }
    } catch (error) {
      console.log(error.response.data);
      this.setState({
        updated: false,
        isLoading: false,
        error: true,
      });
    }
  }

This method fires as soon as the page is reached. It extracts the token from the URL query parameters and passes it back to the server’s reset route to verify the token is legit.

Then, the server either responds with an "a-ok", this token is valid and associated with the user or a "no", the token’s no good anymore for some reason.

The updatePassword() function

 updatePassword = async (e) => {
    e.preventDefault();
    const { username, password } = this.state;
    const {
      match: {
        params: { token },
      },
    } = this.props;
    try {
      const response = await axios.put(
        'http://localhost:3003/updatePasswordViaEmail',
        {
          username,
          password,
          resetPasswordToken: token,
        },
      );
      if (response.data.message === 'password updated') {
        this.setState({
          updated: true,
          error: false,
        });
      } else {
        this.setState({
          updated: false,
          error: true,
        });
      }
    } catch (error) {
      console.log(error.response.data);
    }
  };

This is the function that will fire if the user is authenticated and allowed to reset their password. It also accesses a specific route on the server called updatePasswordViaEmail() (I did this because I gave users a separate route to update their password while logged in to the app, as well), and once the updated password has been saved to the database, a success message response is sent back to the client.

The render() method

  render() {
    const { password, error, isLoading, updated } = this.state;

    if (error) {
      return (
        <div>
          <HeaderBar title={title} />
          <div style={loading}>
            <h4>Problem resetting password. Please send another reset link.</h4>
            <LinkButtons
              buttonText="Go Home"
              buttonStyle={homeButton}
              link="/"
            />
            <LinkButtons
              buttonStyle={forgotButton}
              buttonText="Forgot Password?"
              link="/forgotPassword"
            />
          </div>
        </div>
      );
    }

    if (isLoading) {
      return (
        <div>
          <HeaderBar title={title} />
          <div style={loading}>Loading User Data...</div>
        </div>
      );
    }

    return (
      <div>
        <HeaderBar title={title} />
        <form className="password-form" onSubmit={this.updatePassword}>
          <TextField
            style={inputStyle}
            id="password"
            label="password"
            onChange={this.handleChange('password')}
            value={password}
            type="password"
          />
          <SubmitButtons
            buttonStyle={updateButton}
            buttonText="Update Password"
          />
        </form>

        {updated && (
          <div>
            <p>
              Your password has been successfully reset, please try logging in again.
            </p>
            <LinkButtons
              buttonStyle={loginButton}
              buttonText="Login"
              link="/login"
            />
          </div>
        )}
        <LinkButtons buttonText="Go Home" buttonStyle={homeButton} link="/" />
      </div>
    );
  }

The last piece of this component, is the render() method. Initially, while the token is being verified for its validity, the loading message shows.

If the link is invalid in some way, the error message shows on the screen with links back to the home screen or forgot password page.

If the user is authorized to reset their password, they get the new password input with the updatePassword() function attached to it, and once the server’s responded with success updating the password, the updated boolean is set to true and the Your password has been successfully reset... message is shown along with a login button.

Backend code: reset password & update password

Ok, I’m in the home stretch now. Here’s the last two routes on the server side you’ll need. These correspond to the two methods I just walked through on the client side in the React ResetPassword.js component.

resetPassword.js

/* eslint-disable no-console */
/* eslint-disable max-len */
import Sequelize from 'sequelize';
import User from '../sequelize';

// eslint-disable-next-line prefer-destructuring
const Op = Sequelize.Op;

module.exports = (app) => {
  app.get('/reset', (req, res) => {
    User.findOne({
      where: {
        resetPasswordToken: req.query.resetPasswordToken,
        resetPasswordExpires: {
          [Op.gt]: Date.now(),
        },
      },
    }).then((user) => {
      if (user == null) {
        console.error('password reset link is invalid or has expired');
        res.status(403).send('password reset link is invalid or has expired');
      } else {
        res.status(200).send({
          username: user.username,
          message: 'password reset link a-ok',
        });
      }
    });
  });
};

This is the route that’s called on the componentDidMount() lifecycle method on the client side. It checks the resetPasswordToken passed from the link’s query parameters and date timestamp to ensure that everything’s good.

You’ll notice the resetPasswordExpires parameter has an odd looking $gt: Date.now() parameter. This is an operator alias comparator, which Sequelize allows me to use, and all the $gt: stands for is "greater than" whatever it is being compared to. In this instance, it’s comparing the current time to the expiration time stamp saved to the database when the reset password email was sent, to make sure the password’s being reset less than an hour after the email was sent.

As long as both parameters are valid for that user, the client is sent a successful response and the user can proceed with the password reset.

updatePasswordViaEmail.js

/* eslint-disable no-console */
import bcrypt from 'bcrypt';
import Sequelize from 'sequelize';
import User from '../sequelize';

// eslint-disable-next-line prefer-destructuring
const Op = Sequelize.Op;

const BCRYPT_SALT_ROUNDS = 12;

module.exports = app => {
  app.put('/updatePasswordViaEmail', (req, res) => {
    User.findOne({
      where: {
        username: req.body.username,
        resetPasswordToken: req.body.resetPasswordToken,
        resetPasswordExpires: {
          [Op.gt]: Date.now(),
        },
      },
    }).then(user => {
      if (user == null) {
        console.error('password reset link is invalid or has expired');
        res.status(403).send('password reset link is invalid or has expired');
      } else if (user != null) {
        console.log('user exists in db');
        bcrypt
          .hash(req.body.password, BCRYPT_SALT_ROUNDS)
          .then(hashedPassword => {
            user.update({
              password: hashedPassword,
              resetPasswordToken: null,
              resetPasswordExpires: null,
            });
          })
          .then(() => {
            console.log('password updated');
            res.status(200).send({ message: 'password updated' });
          });
      } else {
        console.error('no user exists in db to update');
        res.status(401).json('no user exists in db to update');
      }
    });
  });
};

This is the second route that’s called when the user submits his/her password to be updated.

Once again, I find the user in the database and double check that the resetPasswordToken also exists and is valid (the username was passed back to the client from the reset route above and held in the app’s state until the update function was called), I hash the new password using my bcrypt module (just like my Passport.js middleware does when a new user is being written into the database initially),update that user’s password in the database with the new hash and set both the resetPasswordToken and resetPasswordExpires columns back to null, so the same link can’t be used more than once.

As soon as that’s complete, the server sends back a 200 response with the success message, Password updated for the client.

And you’ve successfully reset a user’s password via email. Not too tough.

Conclusion

At first glance, resetting a user’s password via an email link seems a bit daunting. But Nodemailer helps make a major factor (the emailing bit) simple. And once that’s done it’s just a few routes on the server side and inputs on the client side to get that password updated for the user.

Check back in a few weeks, I’ll be writing about using Puppeteer and headless Chrome for end-to-end testing or something else related to web development, so subscribe to my free newsletter so you don’t miss out.

Thanks for reading, I hope this gives you an idea of how to use Nodemailer to send password reset emails for a MERN application.

References & Further Resources

The Absolute Easiest Way to Debug Node.js - with VS Code

Sun, 11 Nov 2018 00:00:00 GMT

</figure>

Enter the Pain of Debugging in Node.js

If you’ve ever had the pleasure of writing code for a Node.js project, you know what I’m talking about when I say debugging it to figure out what’s going wrong isn’t the easiest thing.

Unlike JavaScript in the browser, or Java with a powerful IDE like IntelliJ, you can’t just set breakpoints everywhere, refresh the page or restart the compiler and slowly walk through the code examining objects, evaluating functions and finding the mutation or missing variable, etc. You just can’t, and it stinks.

But Node is possible to debug, it just takes a little more elbow grease.

Let’s go over the debugging options and then I’ll show you the easiest way I’ve come across to debug Node.js in my own development.

Options to debug in Node.js

There’s a number of ways to debug your misbehaving Node.js program, I’ve listed them out below with links to learn more if you so desire.

console.log() — the tried and true standby, this one really needs no further explanation if you’ve ever written a line of JavaScript. It’s built in to Node.js and prints in the terminal just like it’s built into JavaScript and prints in the browser’s console.

In Java, it’s System.out.println(), in Python it’s print() - you get the idea. This is the easiest one to implement, and it’s the fastest way to litter your clean code with extra lines of info — but it can also (sometimes) help you find and fix the error.

Node.js documentation —-inspect — The Node.js docs themselves understand debugging isn’t the easiest, so they’ve made a handy reference to help get people started.

It’s useful, but honestly, unless you’ve been programming for a while, it’s not exactly the easiest thing to decipher. They pretty quickly go down the rabbit hole of UUIDs, WebSockets, and security implications and I start to feel in over my head. And thinking to myself: there’s got to be a less complex way to do this.

Chrome DevTools — Paul Irish wrote a great blog post back in 2016 (updated in 2018) about debugging Node.js using Chrome DevTools, and from the looks of things it sounded pretty simple, and like a great step forward for debugging.

Fast forward half an hour, and I still hadn’t managed to connect a DevTools window to my simple Node program and I wasn’t so sure anymore. Maybe I just can’t follow directions, but Chrome DevTools seems to make it more complicated to debug than it should be.

JetBrains — one of my very favorite software development companies and makers of IntelliJ and WebStorm to name just a few IDEs, JetBrains is great. They’ve got a fantastic ecosystem of plugins for their tools and until recently, they were my go-to IDE without question.

With such a dedicated user base came lots of helpful articles like this, which walk through debugging Node, but similar to the Node documentation and the Chrome DevTools options, it’s not easy. You have to create a debugging configuration, attach running processes and do a good bit of configuration in the preferences before WebStorm is ready to go.

Visual Studio Code — this is my new gold standard for Node debugging. I never thought I’d say it, but I am all in on VS Code, and every new feature release the team does, makes me love this IDE more.

VS Code does what every other option for debugging Node.js fails to do, it makes it stupid simple. If you want to get fancier with your debugging you absolutely can, but they break it down to make it simple enough anyone can get up and running quickly, regardless of how familiar you are with IDEs, Node, programming, etc. And it’s awesome.

Setting up VS Code to debug in Node.js

</figure>

Sorry, I couldn’t resist this meme — it’s just so appropriate.

Ok, so let’s walk through setting up VS Code to debug Node. I’ll assume you’ve already downloaded VS Code from the link I posted above, so we’re ready to start setting it up.

In VS Code, open up Preferences > Settings and in the search box type in “node debug”. Under the Extensions tab there should be one extension titled “Node debug”. From here, click the first box: Debug > Node: Auto Attach and set the drop down to “on”. You’re almost ready to go now. Yes, it really is that easy.

<figcaption>Here’s what you should see under the Settings tab. Set the first drop down "Debug > Node: Auto Attach" to "on".</figcaption>

</figure>

Now, go to your Node.js project file, and set some breakpoints by clicking on the left hand side of the file wherever you’d like to see your code stop, and in the terminal type node --inspect <FILE NAME>. Now watch the magic happen…

<figcaption>See the red breakpoints on the lefthand side of the file? See the node — inspect readFileStream.js in the terminal? That’s it.</figcaption>

</figure>

VS Code debugging in action

If you need a Node.js project to test this out with, you can download my repo here. It was made to test different forms of streaming large amounts of data with Node, but it works really well for this demo. If you’d like to see more about streaming data with Node and performance optimization, you can see my posts here and here.

Once you hit Enter, your VS Code terminal should turn orange at the bottom to indicate you’re in debug mode and your console will print some message along the lines of "Debugger Attached".

<figcaption>The orange toolbar and "Debugger attached" message will tell you VS Code is running correctly in debug mode.</figcaption>

</figure>

Once you see this happening, congrats, you’re running in debug mode in Node.js!

Now, you can see your breakpoints in the bottom left corner of the screen (and can toggle them on and off with the checkboxes), and you can step through the code just like you would in a browser with the little play, step over, step in, restart, etc. buttons at the top center of the IDE. VS Code even highlights the breakpoint and line you’ve stopped on with yellow, making it easier to follow along.

<figcaption>Hit the play button at the top to step from one break point to the next one in your code.</figcaption>

</figure>

As you step from breakpoint to breakpoint, you can see the program printing out the console.logs in the debug console at the bottom of the VS Code IDE and the yellow highlighting will travel with you, as well.

<figcaption>As you can see, when we stop on breakpoints, we can see all the local scope info we could explore in the console in the upper left corner of VS Code.</figcaption>

</figure>

As you can see, as I progress through the program, more prints out to the debug console the further through the breakpoints I go, and along the way, I can explore the objects and functions in the local scope using the tools in the upper left hand corner of VS Code, just like I can explore scope and objects in the browser. Nice!

That was pretty painless, huh?

Conclusion

Node.js debugging doesn’t have to be the headache it was in the past, and it doesn’t need to involve 500 console.log()s in the codebase to figure out where the bug is.

Visual Studio Code’s Debug > Node: Auto Attach setting makes that a thing of the past, and I, for one, am so thankful.

Check back in a few weeks, I’ll be writing about end-to-end testing with Puppeteer and headless Chrome or using Nodemailer to reset passwords in a MERN application, so please follow me so you don’t miss out.

Thanks for reading, I hope this gives you an idea of how to more easily and effectively debug your Node.js programs with a little assistance from VS Code.

References & Further Resources

Github, Node Read File Repo
Node.js documentation, inspector
Paul Irish’s blog on using Chrome DevTools to debug Node.js
JetBrains documentation, Running and Debugging Node.js
Visual Studio Code download
VS Code debugging Node.js docs

Streams For the Win: A Performance Comparison of Node.js Methods for Reading Large Datasets (Pt 2)

Wed, 24 Oct 2018 00:00:00 GMT

</figure>

Introduction

If you’ve been keeping up with my writing, a few weeks ago, I published a blog talking about a variety of ways to use Node.js to read really large datasets.

To my surprise, it did exceptionally well with readers — this seemed (to me) like a topic many others have already covered in posts, blogs and forums, but for whatever reason, it got the attention of a lot of people. So, thank you to all of you who took the time to read it! I really appreciate it.

One particularly astute reader: Martin Kock, went so far as to ask how long it took to parse the files. It seemed as if he’d read my mind, because part two of my series on using Node.js to read really, really large files and datasets involves just that.

In this post, I’ll evaluate the three different methods in Node.js I used to read the files, to determine which is most performant.

The challenge from part 1

I won’t go into the specifics of the challenge and solution, because you can read my first post for all the details, but I will give you the high level overview.

A person from a Slack channel I’m a member of, posted a coding challenge he’d received, which involved reading a very large dataset (over 2.5GB in total), parsing through the data and pulling out various pieces of information.

It challenged programmers to print out:

Total lines in the file.
Names in the 432nd and 43243rd indexes.
Counts for total numbers donations per month.
And the most common first name in the files, and how a count of how often that name occurred.

Here's a link to the data if you'd like to view it: https://www.fec.gov/files/bulk-downloads/2018/indiv18.zip

The three different solutions for smaller datasets

As I worked towards my ultimate end goal of processing a large dataset, I came up with three solutions in Node.js.

Solution #1: fs.readFile()

The first involved Node.js’s native method of fs.readFile(), and consisted of reading in the whole file, holding it in memory, and performing the operations on the entire file, then returning the results. At least for smaller files it worked, but when I got to the largest file size, my server crashed with a JavaScript heap out of memory error.

Solution #2: fs.createReadStream() & rl.readLine()

My second solution also involved another couple of methods native to Node.js: fs.createReadStream() and rl.readLine(). In this iteration, the file was streamed through Node.js in an input stream, and I was able to perform individual operations on each line, then cobble all those results together in the output stream.

Again, this worked pretty well on smaller files, but once I got to the biggest file, the same error happened. Although Node.js was streaming the inputs and outputs, it still attempted to hold the whole file in memory while performing the operations (and couldn’t handle the whole file).

Solution #3: event-stream

In the end, I came up with only one solution in Node.js that was able to handle the full 2.55GB file I wanted to parse through at once.

Fun fact: Node.js can only hold up to 1.67GB in memory at any one time, after that, it throws a JavaScript heap out of memory error.

My solution involved a popular NPM package called event-stream, which actually let me perform operations on the throughput stream of data, instead of just the input and output streams, as Node.js’s native capabilities allow.

You can see all three of my solutions here in GitHub.

And I solved the problem, which was my initial goal, but it got me thinking: was my solution really the most performant of the three options?

Comparing solutions to find the optimal one

Now I had a new goal: determine which of my solutions was best.

Since I couldn’t use the full 2.55GB file with the Node.js native solutions, I chose to use one of the smaller files that was about 400MB worth of data, which I’d used for testing while I was developing my solutions.

For performance testing Node.js, I came across two ways to keep track of the file and individual function processing times, and I decided to incorporate both to see how great the differences were between the two methods (and make sure I wasn’t completely off the rails with my timing).

console.time() & console.timeEnd()

Node.js has some handy, built-in methods available to it for timing and performance testing, called console.time() and console.timeEnd(), respectively. To use these methods, I only had to pass in the same label parameter for both time() and timeEnd(), like the code below, and Node’s smart enough to output the time between them after the function’s done.

// timer start
console.time('label1');

// run function doing something in the code
doSomething();

// timer end, where the difference between the timer start and timer end is printed out
console.timeEnd('label1');

// output in console looks like: label1 0.002ms

That’s one method I used to figure out how long it took to process the dataset.

performance-now

The other, tried and well-liked performance testing module I came across for Node.js, is hosted on NPM as performance-now.

7+ million downloads per week from NPM, can’t be too wrong, right??

Implementing the performance-now module into my files was also almost as easy as the native Node.js methods, too. Import the module, set a variable for the start and end of the instantiation of the method, and compute the time difference between the two.

// import the performance-now module at the top of the file
const now = require('performance-now');

// set the start of the timer as a variable
const start = now();

// run function doing something in the code
doSomething();

// set the end of the timer as a variable
const end = now();

// Compute the duration between the start and end
console.log('Performance for timing for label:' + (end — start).toFixed(3) + 'ms';

// console output looks like: Performance for timing label: 0.002ms

I figured that by using both Node’s console.time() and performance-now at the same time, I could split the difference and get a pretty accurate read on how long my file parsing functions were really taking.

Below are code snippets implementing console.time() and performance-now in each of my scripts. These are only snippets of one function each — for the full code, you can see my repo here.

Fs.readFile() code implementation sample

console.time('line count');
let t0 = now();

if(contents === undefined){
  totalLines = contents.split('\n').length - 1;
}
console.log(totalLines);

let t1 = now();
console.timeEnd('line count');

console.log(`Performance now line count timing: ` + (t1 - t0).toFixed(3));

Since this script is using the fs.readFile() implementation, where the entire file is read into memory before any functions are executed on it, this is the most synchronous-looking code.

It’s not actually synchronous, that’s an entirely separate Node method called fs.readFileSync() - this method just resembles it.

But it’s easy to see the total line count of the file and the two timing methods bookending it to determine how long it takes to execute the line count.

Fs.createReadStream() code implementation sample

Input Stream (line-by-line):

console.time('line count');

let t0 = now();

Output Stream (once full file’s been read during input):

let t1 = now();

console.log(lineCount);
console.timeEnd('line count');
console.log(`Performance now line count timing: ` + (t1 - t0).toFixed(3));

As the second solution using fs.createReadStream() involved creating an input and output stream for the file, I broke the code snippets into two separate screenshots, as the first is from the input stream (which is running through the code line by line) and the second is the output stream (compiling all the resulting data).

Event Stream code implementation sample

Throughput Stream (also line-by-line):

console.time('line count');

let t0 = now();

totalLines++;

On Stream End:

let t1 = now();

console.log(totalLines);
console.timeEnd('line count');
console.log(`Performance now line count timing: ` + (t1 - t0).toFixed(3));

The event-stream solution looks pretty similar to the fs.createReadStream(), except instead of an input stream, the data is processed in a throughput stream. And then once the whole file’s been read and all the functions have been done on the file, the stream’s ended and the required information is printed out.

Results

Now on to the moment we’ve all been waiting for: the results!

I ran all three of my solutions against the same 400MB dataset, which contained almost 2 million records to parse through.

<figcaption>Streams for the win!</figcaption>

</figure>

As you can see from the table, fs.createReadStream() and event-stream both fared well, but overall, event-stream has to be the grand winner in my mind, if only for the fact that it can process much larger file sizes than either fs.readFile() or fs.createReadStream().

The percentage improvements are included at the end of the table above as well, for reference.

fs.readFile() just got blown out of the water by the competition. By streaming the data, processing times for the file improved by at least 78% — sometimes close to almost a 100%, which is pretty darn impressive.

Below are the raw screenshots from my terminal for each of my solutions.

Solution #1: fs.readFile()

<figcaption>The solution using only: fs.readFile()</figcaption>

</figure>

Solution #2: fs.createReadStream() & rl.readLine()

<figcaption>The solution using fs.createReadStream() and rl.readLine()</figcaption>

</figure>

Solution #3: event-stream

<figcaption>The solution using event-stream</figcaption>

</figure>

Bonus

Here’s a screenshot of my event-stream solution churning through the 2.55GB monster file, as well. And here’s the time difference between the 400MB file and the 2.55GB file too.

<figcaption>Look at those blazing fast speeds, even as the file size climbs by almost 6X.</figcaption>

</figure>

Solution #3: event-stream (on the 2.55GB file)

<figcaption>Look at those blazing fast speeds, even as the file size climbs by almost 6X.</figcaption>

</figure>

Conclusion

In the end, streams both native to Node.js and not, are way, WAY more efficient at processing large data sets.

Thanks for coming back for part 2 of my series using Node.js to read really, really large files. If you’d like to read the first blog again, you can access it here.

I’ll be back in a couple weeks with a new JavaScript topic — possibly debugging in Node or end-to-end testing with Puppeteer and headless Chrome, so subscribe to my free newsletter so you don't miss out.

Thanks for reading, I hope this gives you an idea of how to handle large amounts of data with Node.js efficiently and performance test your solutions.

References & Further Resources

Github, Read File Repo
Node.js Documentation, File System
Node.js Documentation, Console.Time
NPM, Performance Now
NPM, EventSream
Link to the FEC data

Using Node to Read Really, Really Large Datasets (Pt 1)

Thu, 11 Oct 2018 00:00:00 GMT

</figure>

Introduction

This blog post has an interesting inspiration point. Last week, someone in one of my Slack channels, posted a coding challenge he’d received for a developer position with an insurance technology company.

It piqued my interest as the challenge involved reading through very large files of data from the Federal Elections Commission and displaying back specific data from those files. Since I’ve not worked much with raw data, and I’m always up for a new challenge, I decided to tackle this with Node.js and see if I could complete the challenge myself, for the fun of it.

Here’s the 4 questions asked, and a link to the data set that the program was to parse through.

Write a program that will print out the total number of lines in the file.
Notice that the 8th column contains a person’s name. Write a program that loads in this data and creates an array with all name strings. Print out the 432nd and 43243rd names.
Notice that the 5th column contains a form of date. Count how many donations occurred in each month and print out the results.
Notice that the 8th column contains a person’s name. Create an array with each first name. Identify the most common first name in the data and how many times it occurs.

Link to the data: https://www.fec.gov/files/bulk-downloads/2018/indiv18.zip

When you unzip the folder, you should see one main .txt file that’s 2.55GB and a folder containing smaller pieces of that main file (which is what I used while testing my solutions before moving to the main file).

Not too terrible, right? Seems doable.

Today let’s talk about how I approached parsing this really, really large file using Node.js, and some various solutions I came up with along the way.

The Two Original Node.js Solutions I Came Up With

Processing large files is nothing new to JavaScript, in fact, in the core functionality of Node.js, there are a number of standard solutions for reading and writing to and from files.

The most straightforward is fs.readFile() wherein, the whole file is read into memory and then acted upon once Node has read it, and the second option is fs.createReadStream(), which streams the data in (and out) similar to other languages like Python and Java.

The Solution I Chose to Run With & Why

Since my solution needed to involve such things as counting the total number of lines and parsing through each line to get donation names and dates, I chose to use the second method: fs.createReadStream(). Then, I could use the rl.on(‘line’,...) function to get the necessary data from each line of code as I streamed through the document.

It seemed easier to me then having to split apart the whole file once it was read in and run through the lines that way.

Node.js `createReadStream()` & `readFile()` Code Implementation

Below is the code I came up with using Node.js’s fs.createReadStream() function. I’ll break it down below.

readFileStream.js

var fs = require('fs');
var readline = require('readline');
var stream = require('stream');

var instream = fs.createReadStream('itcont.txt');
var outstream = new stream();
var rl = readline.createInterface(instream, outstream);

//get line count for file
var lineCount = 0;

// create array list of names
var names = [];

// donations occurring in each month
var dateDonationCount = [];
var dateDonations = {};

// list of first names, and most common first name
var firstNames = [];
var dupeNames = {};

rl.on('line', function(line) {
  // increment line count
  lineCount++;

  // get all names
  var name = line.split('|')[7];
  names.push(name);

  // get all first halves of names
  var firstHalfOfName = name.split(', ')[1];
  if (firstHalfOfName !== undefined) {
    firstHalfOfName.trim();
    // filter out middle initials
    if (firstHalfOfName.includes(' ') && firstHalfOfName !== ' ') {
      firstName = firstHalfOfName.split(' ')[0];
      firstName.trim();
      firstNames.push(firstName);
      dupeNames[firstName] = (dupeNames[firstName] || 0) + 1;
    } else {
      firstNames.push(firstHalfOfName);
      dupeNames[firstHalfOfName] = (dupeNames[firstHalfOfName] || 0) + 1;
    }
  }

  // year and month
  var timestamp = line.split('|')[4].slice(0, 6);
  var formattedTimestamp = timestamp.slice(0, 4) + '-' + timestamp.slice(4, 6);
  dateDonationCount.push(formattedTimestamp);
  dateDonations[formattedTimestamp] =
    (dateDonations[formattedTimestamp] || 0) + 1;
});

rl.on('close', function() {
  // total line count
  console.log(lineCount);

  // names at various points in time
  console.log(names[432]);
  console.log(names[43243]);

  // most common first name
  var sortedDupeNames = Object.entries(dupeNames);

  sortedDupeNames.sort((a, b) => {
    return b[1] - a[1];
  });
  console.log(sortedDupeNames[0]);

  const name = sortedDupeNames[0][0];
  const nameOccurrences = sortedDupeNames[0][1];
  console.log(
    `The most common name is '${name}' with ${nameOccurrences} occurrences.`,
  );

  // number of donations per month
  const logDateElements = (key, value, map) => {
    console.log(
      `Donations per month and year: ${value} and donation count ${key}`,
    );
  };
  new Map(Object.entries(dateDonations)).forEach(logDateElements);
});

The very first things I had to do to set this up, were import the required functions from Node.js: fs (file system), readline, and stream. These imports allowed me to then create an instream and outstream and then the readLine.createInterface(), which would let me read through the stream line by line and print out data from it.

I also added some variables (and comments) to hold various bits of data: a lineCount, names array, donation array and object, and firstNames array and dupeNames object. You’ll see where these come into play a little later.

Inside of the rl.on('line',...) function, I was able to do all of my line-by-line data parsing. In here, I incremented the lineCount variable for each line it streamed through. I used the JavaScript split() method to parse out each name and added it to my names array. I further reduced each name down to just first names, while accounting for middle initials, multiple names, etc. along with the first name with the help of the JavaScript trim(), includes(), and split() methods. And I sliced the year and date out of date column, reformatted those to a more readable YYYY-MM format, and added them to the dateDonationCount array.

In the rl.on('close',...) function, I did all the transformations on the data I’d gathered into arrays and console.logged out all my data for the user to see.

The lineCount and names at the 432nd and 43,243rd index, required no further manipulation. Finding the most common name and the number of donations for each month was a little trickier.

For the most common first name, I first had to create an object of key value pairs for each name (the key) and the number of times it appeared (the value), then I transformed that into an array of arrays using the ES6 function Object.entries(). From there, it was a simple task to sort the names by their value and print the largest value.

Donations also required me to make a similar object of key value pairs, create a logDateElements() function where I could nicely using ES6’s string interpolation to display the keys and values for each donation month. And then create a new Map() transforming the dateDonations object into an array of arrays, and looping through each array calling the logDateElements() function on it. Whew! Not quite as simple as I first thought.

But it worked. At least with the smaller 400MB file I was using for testing…

After I’d done that with fs.createReadStream(), I went back and also implemented my solutions with fs.readFile(), to see the differences. Here’s the code for that, but I won’t go through all the details here — it’s pretty similar to the first snippet, just more synchronous looking (unless you use the fs.readFileSync() function, though, JavaScript will run this code just as asynchronously as all its other code, not to worry.

readFile.js

var fs = require('fs');

var totalLines = 0;
var lines = [];
var names = [];
var firstNames = [];
var dupeNames = {};
var dateDonationCount = [];
var dateDonations = {};

fs.readFile('itcont.txt', 'utf8', (err, contents) => {
  if (contents !== undefined) {
    totalLines = contents.split('\n').length - 1;
  }
  console.log(totalLines);

  if (contents !== undefined) {
    lines = contents.split('\n');
    lines.forEach(line => {
      var name = line.split('|')[7];
      names.push(name);
    });
  }

  console.log(names[432]);
  console.log(names[43243]);

  names.forEach(name => {
    var firstHalfOfName = name.split(', ')[1];
    if (firstHalfOfName !== undefined) {
      firstHalfOfName.trim();

      if (firstHalfOfName !== ' ' && firstHalfOfName.includes(' ')) {
        firstName = firstHalfOfName.split(' ')[0];
        firstName.trim();
        firstNames.push(firstName);
      } else {
        firstNames.push(firstHalfOfName);
      }
    }
  });

  firstNames.forEach(x => {
    dupeNames[x] = (dupeNames[x] || 0) + 1;
  });
  var sortedDupeNames = [];
  sortedDupeNames = Object.entries(dupeNames);

  sortedDupeNames.sort((a, b) => {
    return b[1] - a[1];
  });
  console.log(sortedDupeNames[0]);

  lines.forEach(line => {
    var timestamp = line.split('|')[4].slice(0, 6);
    var formattedTimestamp =
      timestamp.slice(0, 4) + '-' + timestamp.slice(4, 6);
    dateDonationCount.push(formattedTimestamp);
  });

  dateDonationCount.forEach(x => {
    dateDonations[x] = (dateDonations[x] || 0) + 1;
  });
  
  const logDateElements = (key, value, map) => {
    console.log(
      `Donations per month and year: ${value} and donation count ${key}`,
    );
  };
  new Map(Object.entries(dateDonations)).forEach(logDateElements);
});

If you’d like to see my full repo with all my code, you can see it here.

Initial Results: Round 1

With my working solution, I added the file path into readFileStream.js file for the 2.55GB monster file, and watched my Node server crash with a JavaScript heap out of memory error.

<figcaption>Failed. Whomp, whomp...</figcaption>

</figure>

As it turns out, although Node.js is streaming the file input and output, in between, it is still attempting to hold the entire file contents in memory, which it can’t do with a file that size. Node can hold up to 1.5GB in memory at one time, but no more.

So neither of my current solutions was up for the full challenge.

I needed a new solution. A solution for even larger datasets running through Node.

The New Data Streaming Solution

I found my solution in the form of EventStream, a popular NPM module with over 2 million weekly downloads and a promise “to make creating and working with streams easy”.

With a little help from EventStream’s documentation, I was able to figure out how to, once again, read the code line by line and do what needed to be done, hopefully, in a more CPU friendly way to Node.

EventStream Code Implementation

Here’s my code new code using the NPM module EventStream.

readFileEventStream.js

var fs = require('fs');
var es = require('event-stream');

var totalLines = 0;
var names = [];
var firstNames = [];
var dupeNames = {};
var dateDonationCount = [];
var dateDonations = {};

var s = fs
  .createReadStream('itcont.txt')
  .pipe(es.split())
  .pipe(
    es
      .mapSync(function(line) {
        totalLines++;

        // get all names
        var name = line.split('|')[7];
        if (totalLines === 433 || totalLines === 43244) {
          names.push(name);
        }

        // get all first halves of names
        var firstHalfOfName = name.split(', ')[1];

        if (firstHalfOfName !== undefined) {
          firstHalfOfName.trim();

          // filter out middle initials
          if (firstHalfOfName.includes(' ') && firstHalfOfName !== ' ') {
            firstName = firstHalfOfName.split(' ')[0];
            firstName.trim();
            firstNames.push(firstName);
            dupeNames[firstName] = (dupeNames[firstName] || 0) + 1;
          } else {
            firstNames.push(firstHalfOfName);
            dupeNames[firstHalfOfName] = (dupeNames[firstHalfOfName] || 0) + 1;
          }
        }

        // year and month
        var timestamp = line.split('|')[4].slice(0, 6);
        var formattedTimestamp =
          timestamp.slice(0, 4) + '-' + timestamp.slice(4, 6);
        dateDonationCount.push(formattedTimestamp);
        dateDonations[formattedTimestamp] =
          (dateDonations[formattedTimestamp] || 0) + 1;
      })
      .on('error', function(err) {
        console.log('Error while reading file.', err);
      })
      .on('end', function() {
        console.log('Read entire file.');
        console.log(totalLines);

        sortedDupeNames = Object.entries(dupeNames);

        sortedDupeNames.sort((a, b) => {
          return b[1] - a[1];
        });
        console.log(sortedDupeNames[0]);

        const name = sortedDupeNames[0][0];
        const nameOccurrences = sortedDupeNames[0][1];
        console.log(
          `The most common name is '${name}' with ${nameOccurrences} occurrences.`,
        );

        // number of donations per month
        const logDateElements = (key, value, map) => {
          console.log(
            `Donations per month and year: ${value} and donation count ${key}`,
          );
        };
        new Map(Object.entries(dateDonations)).forEach(logDateElements);
      }),
  );

The biggest change was the pipe commands at the beginning of the file — all of that syntax is the way EventStream’s documentation recommends you break up the stream into chunks delimited by the \n character at the end of each line of the .txt file.

The only other thing I had to change was the names answer. I had to fudge that a little bit since if I tried to add all 13MM names into an array, I again, hit the out of memory issue. I got around it, by just collecting the 432nd and 43,243rd names and adding them to their own array. Not quite what was being asked, but hey - I had to get a little creative.

Results from Node.js & EventStream: Round 2

Ok, with the new solution implemented, I again, fired up Node.js with my 2.55GB file and my fingers crossed this would work. Check out the results.

</figure>

Success!

Conclusion

In the end, Node.js’s pure file and big data handling functions fell a little short of what I needed, but with just one extra NPM package, EventStream, I was able to parse through a massive dataset without crashing the Node server.

Stay tuned for part two of this series where I compare my three different ways of reading data in Node.js with performance testing to see which one is truly superior to the others. The results are pretty eye opening — especially as the data gets larger…

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading, I hope this gives you an idea of how to handle large amounts of data with Node.js.

References & Further Resources

Node.js Documentation, File System
Node.js Documentation, Readline
Github, Read File Repo
NPM, EventSream

How to Use Netflix's Eureka and Spring Cloud for Service Registry

Sun, 30 Sep 2018 00:00:00 GMT

</figure>

Introduction

One of the main tenets of the microservice architecture pattern is that a set of loosely-coupled, collaborating services works together to form a cohesive, whole application. In this way, it’s not so much about the individual services, it’s more about ensuring the interactions between the services remains reliable and failure-tolerant.

What this means in practice, is removing hard-coded information and replacing it with dynamically updating environment variables, having separate databases for each service, removing as much dependency between services as possible (so if necessary, a service that needs more instances can easily scale without impacting its counterparts), and generally decreasing the complexity of managing the whole by increasing the complexity of each individual service.

It sounds like a good strategy, but keeping track of all the smaller pieces of the whole makes it a burden on the client to manage all of this.

Which brings me to the topic of my blog today: the Netflix Eureka service registry — and how to use it with Spring Cloud to more effectively manage the complex task of keeping track of services.

As the Github repo for Netflix’s Eureka registry says itself:

“Eureka is a REST (Representational State Transfer) based service that is primarily used in the AWS cloud for locating services for the purpose of load balancing and failover of middle-tier servers.”

— Netflix: Eureka at a Glance, Github

Sounds good? Good. Without further ado, let’s get to setting up this Eureka service registry and a couple of services to see it in practice.

Set Up the Eureka Server

Setting up a Spring-based Eureka server is actually pretty simple. Spring.io, itself, has a great walkthrough here, that helped get me up and running quickly with both a Eureka server and a sample Spring Boot project.

I won’t go through it step-by-step, as you can do that with the tutorial I linked to, or you can download my working code example here. But I will highlight the important things to include for the server setup.

NOTE: All the code snippets shown below have file names linked to the actual GitHub files in my example repo. Feel free to click on them to see the working code in its entirety.

`build.gradle` File

ext {
	springCloudVersion = 'Finchley.SR1'
}

dependencies {
	compile('org.springframework.cloud:spring-cloud-starter-netflix-eureka-server')
	testCompile('org.springframework.boot:spring-boot-starter-test')
}

dependencyManagement {
	imports {
		mavenBom "org.springframework.cloud:spring-cloud-dependencies:${springCloudVersion}"
	}
}

repositories {
	maven {
		url 'https://repo.spring.io/libs-milestone'
	}
}

Inside of your build.gradle file, you must include the dependencies above:

the spring-cloud-starter-netflix-eureka-server
and the spring-cloud-dependencies.

With that done, next up is the application.yml file in the resources folder.

`application.yml` File

spring:
    application:
        name: eureka-service

server:
    port: 8761

eureka:
    client:
        register-with-eureka: false
        fetch-registry: false
        serviceUrl:
            defaultZone: http://eureka-service:8761/eureka

logging:
    level:
        com:
            netflix:
                eureka: OFF
                discovery: OFF

This file doesn’t need much, either. It needs a server port specified so the service doesn’t automatically start on port 8080, which would then conflict with our other Spring Boot client services, when running locally.

A couple more configurations are specified as well, for convenience. The Eureka client is instructed not to register itself upon start up: eureka.client.register-with-eureka: false, and it is told not to search for other registry nodes to connect to, as there are none (at least not while running locally ): eureka.client.fetch-registry: false.

A default URL is listed and the verbose logging for Eureka and any subsequent discovery of services is also turned off.

Right, now there’s just one final step on the main class path.

`EurekaServerApp.java` File

package eurekaServer;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.netflix.eureka.server.EnableEurekaServer;

@EnableEurekaServer
@SpringBootApplication
public class EurekaServerApplication {

	public static void main(String[] args) {
		SpringApplication.run(EurekaServerApplication.class, args);
	}
}

The only addition the main file needs is the @EnableEurekaServer annotation, which tells the Spring Boot service to enable the server. Easy enough.

Ok, we can move on to the services now — this is where things start to get more interesting.

Set Up a Spring Boot Java Service for Eureka to Register

Once again, the Spring.io walkthrough here, does a very good job of setting up the first Spring Boot server and client project, so I’ll just hit the highlights again of what you must include.

dependencyManagement {
	imports {
		mavenBom 'org.springframework.cloud:spring-cloud-dependencies:Finchley.SR1'
	}
}

repositories {
	maven {
		url 'https://repo.spring.io/libs-milestone'
	}
}

dependencies {
	compile('org.springframework.cloud:spring-cloud-starter-netflix-eureka-client')
	compile('org.springframework.boot:spring-boot-starter-web')
	testCompile('org.springframework.boot:spring-boot-starter-test')
	testCompile('org.springframework.cloud:spring-cloud-starter-eurekaServer')
}

For the client service’s build.gradle file, just be sure to include the spring-cloud-starter-netflix-eureka-client, and the spring-cloud-dependencies.

The spring-boot-starter-web dependencies are included so a web endpoint can be created in the service that will show us what the service’s information relative to the Eureka server looks like.

`bootstrap.yml` & `application.yml` Files

bootstrap.yml

spring:
    application:
        name: a-java-service

Once again, some minor configurations are needed for the boostrap.yml and application.yml files. The bootstrap.yml is pictured above, it is picked up before the application.yml file by Spring, so this is where we set the service’s name.

I chose the very original spring.application.name: a-java-service, for this project. That’s how the Eureka server will reference it going forward.

application.yml

server:
    port: 8091

eureka:
    client:
        serviceUrl:
            defaultZone: http://localhost:8761/eureka

And then the application.yml file has configs similar to the Eureka server setup. A port number and the default URL for the Eureka server.

EurekaClientApplication.java File

package eurekaClient;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.ServiceInstance;
import org.springframework.cloud.client.discovery.DiscoveryClient;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.List;

@EnableDiscoveryClient
@SpringBootApplication
public class EurekaClientApplication {

	public static void main(String[] args) {
		SpringApplication.run(EurekaClientApplication.class, args);
	}
}

@RestController
class ServiceInstanceRestController {

	@Autowired
	private DiscoveryClient discoveryClient;

	@RequestMapping("/service-instances/{applicationName}")
	public List<ServiceInstance> serviceInstancesByApplicationName(
			@PathVariable String applicationName) {
		return this.discoveryClient.getInstances(applicationName);
	}
}

Finally, the service is annotated in the main class path file with @EnableDiscoveryClient, which tells the Spring Boot service to activate the Netflix Eureka DiscoveryClient implementation and register its own host and port with the Eureka server.

The REST endpoint defined below on the REST Controller can be used to see all the service instances registered in the Eureka registry at the local URL: http://localhost:8091/service-instances/a-java-service.

The JSON info for the service looks like this:

</figure>

Set Up a Node Service for Eureka to Register

Now, that's a good start, but to be more real-world, I decided I wanted to also register a Node.js service to the Eureka server as well, and that proved a little more challenging.

Luckily, there’s a handy little NPM package out there for just such needs, called eureka-js-client, which is billed as a:

JavaScript implementation of a client for Eureka, the Netflix OSS service registry.

The documentation must be read through to the end because there’s some special gotchas when using the Spring implementation with Eureka, but I managed to get it working with some trial and error.

Here’s what you need to get a sample Node.js project up and running.

package.json

  "dependencies": {
    "eureka-js-client": "^4.4.1",
    "express": "^4.16.3",
    "nodemon": "^1.18.4"
  }

For this Node.js service, we just need to install the eureka-js-client, and I added express and nodemon so I could easily make some REST calls to the application, as well as have the Node server automatically reload as I made tweaks to the system.

server.js

'use strict';

const express = require('express');
const Eureka = require('eureka-js-client').Eureka;

// Constants
const PORT = 3000;
const HOST = '0.0.0.0';
const app = express();

// example configuration
const client = new Eureka({
  // application instance information
  instance: {
    app: 'a-node-service',
    hostName: 'localhost',
    ipAddr: '127.0.0.1',
    statusPageUrl: 'http://localhost:3000',
    vipAddress: 'a-node-service',
    port: {
      $: PORT,
      '@enabled': 'true',
    },
    dataCenterInfo: {
      '@class': 'com.netflix.appinfo.InstanceInfo$DefaultDataCenterInfo',
      name: 'MyOwn',
    },
    registerWithEureka: true,
    fetchRegistry: true,
  },
  eureka: {
    // eureka server host / port
    host: 'localhost',
    port: 8761,
    servicePath: '/eureka/apps/',
  },
});

client.logger.level('debug');
client.start(error => {
  console.log(error || 'NodeJS Eureka Started!');

  // App
  app.get('/', (req, res) => {
    res.send('Hello from NodeJS Eureka Client\n');
    res.end();
  });
});  

app.listen(PORT, HOST);
console.log(`Running on http://${HOST}:${PORT}`);

I used just one server.js file for convenience and because this project is so small to begin with.

What’s important is the new Eureka() set up. Whereas with the Spring Boot projects, we set the application configurations in the bootstrap.yml and application.yml, for the JavaScript project, this config is set in the server.js file or injected with a config file (for larger projects or projects with multiple different configurations based on production lifecycle).

In here is where the application name is defined: app: 'a-node-service', the host name, IP address, port, data center info, etc. are defined.

Every parameter up to the registerWithEureka and fetchRegistry options are required or else the server will crash, but the vipAddress and dataCenterInfo fields can be added exactly as they’re written in the eureka-client-js documentation. They just have to be filled out.

Last, but not least, once the client service has been set up, it is started with the client.start() command, where I am console logging either the error message if it fails to start or the ‘Node.js Eureka Started’ message if connecting to the Eureka registry is successful.

Run the Eureka Service Registry

Let’s get this service registry up and running now. I’ve got full details to run all the service in the README.md of my example project here, but here it is as well.

We’ll cd into each service contained within the one master project folder, if you have a file structure similar to the one prescribed by the Sping.io starter tutorial. Here’s what mine looks like for reference:

root/
├── eureka-client-java/ 
| ├── build.gradle 
| ├── src/ 
| | | ├── main/ 
| | | | ├── java/ 
| | | | ├── resources/ 
| | | | | ├── application.yml 
| | | | | ├── bootstrap.yml
├── eureka-client-node/ 
| ├── server.js
| ├── package.json
| ├── node-modules/
├── eureka-service/
| ├── build.gradle 
| ├── src/ 
| | | ├── main/ 
| | | | ├── java/ 
| | | | ├── resources/ 
| | | | | ├── application.yml

As I said, cd into each repo eureka-client-java/, eureka-client-node/, and eureka-service/, and for the two Spring Boot projects run gradle clean build and then gradle bootRun from the command line.

For the Node.js project, run npm start from another terminal window.

Give it a minute for all the projects to spin up and be found by the registry service, then go to http://localhost:8761 and this is what you should see.

<figcaption>If you look closely in the ‘Instances currently registered with Eureka’ section, you’ll see our applications.</figcaption>

</figure>

This is the Eureka server homepage, and if you look in the second section down the page, you’ll see ‘Instance currently registered with Eureka’, and the two service we built registered with it.

To verify these services are really running, you can go to http://localhost:8091/service-instances/a-java/service and http://localhost:3000/ to see info from each service.

Now, let’s take it a step further, because the whole point of having a service registry is to make it easier for clients registered there to ask questions about how to connect and communicate with the other available services:

“Each service registers itself with the service registry and tells the registry where it lives (host, port, node name) and perhaps other service-specific metadata — things that other services can use to make informed decisions about it. Clients can ask questions about the service topology (“are there any ‘fulfillment-services’ available, and if so, where?”) and service capabilities (“can you handle X, Y, and Z?”).”

— Spring, Microservice Registration and Discovery with Spring Cloud and Netflix’s Eureka

Set Up Extra Endpoints for the Services to Communicate

The next step I wanted to take was making it so I could get the service instance JSON data supplied by the Java service available through an endpoint from the Node.js service — which it can only get by communicating with the info supplied by the Eureka service registry.

Here’s what to do:

server.js

var javaInstance = '';

client.logger.level('debug');
client.start(error => {
  console.log(error || 'NodeJS Eureka Started!');

  javaInstance = client.getInstancesByAppId('A-JAVA-SERVICE');
  // console.log(javaInstance);

  // App
  app.get('/', (req, res) => {
    res.send('Hello from NodeJS Eureka Client\n');
    res.end();
  });

  const javaUrl = `${javaInstance[0].hostName}:${
    javaInstance[0].port.$
  }/service-instances/${javaInstance[0].app}`;

  console.log(javaUrl);

  // get java service info endpoint
  app.get(`/serviceInfo/${javaUrl}`, (req, res) => {
    res.send(JSON.stringify(javaInstance), null, 2);
    res.end();
  });
});

Right before the client.start() command in the server.js file, I defined a variable called javaInstance, and then used the method provided by the eureka-client-js plugin getInstancesByAppId() calling the Spring Boot service registered as a-java-service to get its service information.

With this, I could reconstruct the URL information supplied by the Spring Boot REST endpoint to get that service’s information. In short, I was able to parse out all the necessary info from the Java instance supplied by the Eureka server to reach it through the Node.js endpoint.

Here’s what the URL comes out to be once the host name, port number and app name are acquired: http://localhost:3000/serviceInfo/192.168.1.18:8091/service-instances/A-JAVA-SERVICE .

And here’s what you’d see if both the Node.js and Java services are running and registered with the Eureka server:

<figcaption>Spring Boot service data courtesy of a Node.js service endpoint.</figcaption>

</figure>

Voila. The above JSON (not well formatted) is for the a-java-service, accessed through an endpoint on the Node.js service. Just be sure to make that URL call to the Spring Boot service inside of the client.start() command, otherwise you won’t be able to access all the instance information supplied from the getInstancesByAppId() call to the Eureka server.

At this point, Eureka is doing what it’s designed to do: it registers services and provides information from one to the other, minus the hardcoded complexity developers used to have to be responsible for remembering and implementing manually.

Conclusion

As I said at the beginning, building applications according to the microservice architecture pattern means more complexity managing the individual services, but greater fault tolerance and reliability of the application as a whole.

Service registries like Netflix’s Eureka server help manage this additional complexity, and when coupled with Spring’s Cloud technology it becomes infinitely easier to use. The next step after this would be using something like Zuul to assist with dynamic routing, load balancing between instances, handling security, and more. But that’s for another blog post.

Thanks for reading, I hope this proves helpful and gives you a better understanding of how to use Netflix’s Eureka server with the help of Spring Cloud, and discover Java and Node.js services.

References & Further Resources

Implementing JSON Web Tokens & Passport.js in a React App

Wed, 12 Sep 2018 00:00:00 GMT

</figure>

Introduction

Before you say it, I know, I know. There’s already a literal ton of MERN tutorials out there showing how to use JSON Web Tokens (JWT, pronounced "jot") and Passport.js with Express.

But here’s what every one of those tutorials failed to mention, and what mine will cover: how and why to use the various authentication flavors Passport offers (including passport-jwt), and the gotchas that tripped me up for hours on end as I put together a user registration application.

I don’t know what your experience has been, but the majority of the projects I’ve worked on professionally have come with authorization already prebuilt by someone else. So in addition to improving my MERN skills, I viewed this as an opportunity to learn more about security and authentication and different ways to do it.

This post will mainly cover how to implement of Passport and JWT on the server side, because that’s where most of the magic happens, but if you want, you can see the source files and download the full MERN project here.

Once it’s correctly hooked up on the backend, the frontend is purely React.

Before I jump into all that though, let me give you a quick run down of JWT and Passport.js authentication.

What is a JSON Web Token?

</figure>

A JSON Web Token, according to the site is:

“…an open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between parties as a JSON object.” — JWT.io

Essentially, JWTs are digitally signed tokens that can be verified and trusted, and they’re becoming more and more popular for security and authorization between parties (like servers and clients) and for exchanging sensitive information, while verifying the token hasn’t been tampered with or decoded.

I won’t go into the details of how the tokens work, you can read all about that from a much more qualified source than I am here, but suffice it to say, I chose to use JWT as part of my user verification strategy along with Passport.js to make my simple user registration application secure (and because I was curious to see if I could do it, since I didn’t have a whole lot of hands on experience implementing authorization).

With that covered, the next piece of my authentication solution is Passport.js.

What is Passport.js & Why Should I Use it?

</figure>

Passport.js is:

Simple, unobtrusive authentication for Node.js — Passport.js

And it works exceedingly well with Express.js. Passport is authentication middleware for Node, which serves one purpose, to authenticate requests, in a modular way that leaves all other functionality to the application itself, making code cleaner, easier to maintain and provides a clear separation of concerns.

If you type the term "JavaScript authentication middleware" or even just "JavaScript authentication" into the Google search bar, Passport.js ranks within the top 5 search results. That’s how ubiquitous this solution is in the JavaScript ecosystem.

Did I mention it boasts more than 500+ authentication strategies? It does. Whether you wish to log in with a simple username and password, with Github credentials, Facebook, oAuth, etc., there’s probably a Passport.js strategy for it.

So it was a simple decision to choose Passport as part of my authorization strategy.

I’d like to add documentation on the site is pretty good, though some things like custom callbacks require much more careful reading (and trial and error, for me) to correctly set up. But I’m getting ahead of myself, I’ll cover the gotchas in implementation.

How Do I Implement Them in Express.js (and a little React)?

Now on to the fun part, how do you implement both Passport.js and JWT into an Express/Node application? To be honest, it stumped me for a good while. But after numerous tutorials, re-readings of documentation and turning to Stack Overflow for help, I got to a place of understanding and satisfaction.

I should say, one of my goals for this app was to make it modular, make it super simple to add or remove functionality, so when you see my file structure, you’ll notice all the routes for the CRUD functionality (create, read, update, delete) are built into separate files, and all the Passport authentication is handled in one, centralized file as well. Here’s a look at the file structure of the API portion of my app.

The API File Tree

root/
├── api/ 
| ├── config 
| | ├── passport.js 
| | ├── jwtConfig.js
| ├── server.js 
| ├── sequelize.js 
| ├── package.json
| ├── models/ 
| | ├── user.js
| ├── routes/ 
| | ├── deleteUser.js 
| | ├── findUsers.js 
| | ├── loginUser.js 
| | ├── registerUser.js
| | ├── updateUser.js
| ├── node-modules/

The `package.json` File

And here is the package.json and its dependencies so you can see exactly what I’m using.

package.json

{
  "name": "mysqlregistration-api",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "start": "nodemon server.js --exec babel-node --presets es2015"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/paigen11/mysqlRegistration.git"
  },
  "author": "Paige Niedringhaus",
  "license": "MIT",
  "bugs": {
    "url": "https://github.com/paigen11/mysqlRegistration/issues"
  },
  "homepage": "https://github.com/paigen11/mysqlRegistration#readme",
  "dependencies": {
    "babel-cli": "^6.26.0",
    "babel-preset-es2015": "^6.24.1",
    "babel-preset-stage-0": "^6.24.1",
    "bcrypt": "^3.0.2",
    "body-parser": "^1.18.3",
    "cors": "^2.8.4",
    "dotenv": "^6.1.0",
    "express": "^4.16.3",
    "helmet": "^3.20.1",
    "jsonwebtoken": "^8.3.0",
    "morgan": "^1.9.0",
    "mysql2": "^1.5.3",
    "nodemailer": "^4.6.8",
    "nodemon": "^1.18.3",
    "passport": "^0.4.0",
    "passport-jwt": "^4.0.0",
    "passport-local": "^1.0.0",
    "sequelize": "^5.3.0",
    "sequelize-cli": "^4.0.0",
    "swagger-jsdoc": "^3.2.9",
    "swagger-ui-express": "^4.0.2"
  }
}

The dependencies include a few extras like babel so I can use ES6 syntax in my Node.js app, bcrypt for password hashing and sequelize as my MySQL ORM, but the things you need to focus on are jsonwebtoken, passport, passport-local and passport-jwt. These are the necessities for this blog.

The `server.js` File

I’ll start with the server.js file first, as it requires the least explanation. This file is purely for starting the server, initializing the use of Passport in the app and setting up the routes and parsing of requests from the client side.

server.js

import express from 'express';
import Cors from 'cors';
import bodyParser from 'body-parser';
import logger from 'morgan';
import passport from 'passport';
import helmet from 'helmet';

const app = express();

const API_PORT = process.env.API_PORT || 3000;

require('./config/passport');

app.use(Cors());
app.use(bodyParser.urlencoded({ extended: false }));
app.use(bodyParser.json());
app.use(logger('dev'));
app.use(helmet());
app.use(passport.initialize());

require('./routes/loginUser')(app);
require('./routes/registerUser')(app);
require('./routes/findUsers')(app);
require('./routes/deleteUser')(app);
require('./routes/updateUser')(app);

// eslint-disable-next-line no-console
app.listen(API_PORT, () => console.log(`Listening on port ${API_PORT}`));

module.exports = app;

A little ES6 in the imports and consts, but what you should focus on is the passport.initialize() and the routes.

Pretty straightforward right? Great. Moving on.

You might also have noticed the require('./config/passport');, this is the next step I’ll go over. The JWT and Passport configuration, inside of the folder named config/.

The `jwtConfig.js` File

The JWT config is very simple, it’s the secret required by JWT to encode and decode the tokens. Typically, this would be stored as an environmental variable in a file that’s not checked in to Github, but to show how this works, I’ve set it in here.

jwtConfig.js

module.exports = {
  secret: 'jwt-secret',
};

The `passport.js` File

passport.js

import bcrypt from 'bcrypt';
import jwtSecret from './jwtConfig';

const BCRYPT_SALT_ROUNDS = 12;

const passport = require('passport');
const LocalStrategy = require('passport-local').Strategy;
const JWTstrategy = require('passport-jwt').Strategy;
const ExtractJWT = require('passport-jwt').ExtractJwt;
const User = require('../sequelize');

passport.use(
  'register',
  new LocalStrategy(
    {
      usernameField: 'username',
      passwordField: 'password',
      passReqToCallback: true,
      session: false,
    },
    (req, username, password, done) => {
      try {
        User.findOne({
          where: {
            [Op.or]: [
              {
                username,
              },
              { email: req.body.email },
            ],
          },
        }).then(user => {
          if (user != null) {
            console.log('username or email already taken');
            return done(null, false, {
              message: 'username or email already taken',
            });
          }
          bcrypt.hash(password, BCRYPT_SALT_ROUNDS).then(hashedPassword => {
            User.create({
              username,
              password: hashedPassword,
              email: req.body.email,
            }).then(user => {
              console.log('user created');
              return done(null, user);
            });
          });
        });
      } catch (err) {
        return done(err);
      }
    },
  ),
);

passport.use(
  'login',
  new LocalStrategy(
    {
      usernameField: 'username',
      passwordField: 'password',
      session: false,
    },
    (username, password, done) => {
      try {
        User.findOne({
          where: {
            username,
          },
        }).then(user => {
          if (user === null) {
            return done(null, false, { message: 'bad username' });
          }
          bcrypt.compare(password, user.password).then(response => {
            if (response !== true) {
              console.log('passwords do not match');
              return done(null, false, { message: 'passwords do not match' });
            }
            console.log('user found & authenticated');
            return done(null, user);
          });
        });
      } catch (err) {
        done(err);
      }
    },
  ),
);

const opts = {
  jwtFromRequest: ExtractJWT.fromAuthHeaderWithScheme('JWT'),
  secretOrKey: jwtSecret.secret,
};

passport.use(
  'jwt',
  new JWTstrategy(opts, (jwt_payload, done) => {
    try {
      User.findOne({
        where: {
          id: jwt_payload.id,
        },
      }).then(user => {
        if (user) {
          console.log('user found in db in passport');
          done(null, user);
        } else {
          console.log('user not found in db');
          done(null, false);
        }
      });
    } catch (err) {
      done(err);
    }
  }),
);

Yes, all this is part of the passport.js file.

I realize this is a lot of code to take in. If you’d like to see just the code for Passport and the routes with Passport authentication, I created a few gists here with the code.

I’ll walk through the two types of Passport implementation happening here: passport-local for the register and login methods and passport-jwt for the jwt method.

passport-local uses a username and password, and passport-jwt uses a JWT payload to verify the user is legit.

Like I said earlier, the documentation for both using just Express.js is fairly good, but the trick comes in combining it with the MySQL database and the information passed from the React client on the frontend.

Once the passport-local strategy has been passed a username and password (which I verify that both inputs are at least filled on the client side before ever calling back to the server), the first check I do on the backend is with Sequelize, my SQL ORM (like Mongoose for MongoDB), to determine if that username exists in the database. If it returns null, the authentication fails (no user in the db matches), and typically, a 401 Unauthorized HTTP status would be thrown back from the server to the client with no further information.

Passport.js Gotcha #1: Info in Error Handling

This type of error handling, to me, was the most frustrating thing about Passport. The lack of information to let a user (or myself) know what the actual error was beyond 401 Unauthorized.

But there’s a better solution that requires a little extra work, with custom callbacks. I’ll go into more detail about the callback soon, but for now, you can see if there’s some sort of error, instead of returning a return done(null, user);, Passport can pass back return done(null, false, { message: 'bad username or passwords don't match' });.

This message can then be passed back from the server to the client, actually telling the user what the problem is (which is what I wanted for this app). If you prefer just to tell the user their authentication failed, but not disclose why, that’s cool too. Your choice, but I want to show how error messaging can be done.

The rest of the passport-local strategy is pretty self-explanatory, on register the user’s password is hashed and salted with the encryption package bcrypt, and then when the login method is invoked, it hashes the newly entered password and checks the passwords with bcrypt’s compare function before returning either a positive or negative on the verification.

Now that brings me to the passport-jwt strategy, which is named jwt. This is the authentication that’s called on the protected routes in the application: findUsers, updateUser, deleteUser.

For the jwt strategy, the JWT is passed back from the client with each call to the server (I passed mine in an authorization header with the key: JWT), which is extracted and then decoded using the secret (which is stored in another file, but should really be an environment variable known only to the system).

Once the JWT payload is decrypted, the ID (which for me is the username) can be searched for in the database just like the passport-local strategy does, and either returned with done(null, user);, if the user is found or with done(null, false); if it’s not (which should almost never happen because the JWT includes the username in its encrypted form, so unless that’s somehow been tampered with or the db has been, it should be able to find the user).

Passport.js Gotcha #2: Passport-Local Wants Return, Passport-JWT Does Not

This brings me to my second gotcha which tripped me up for a good bit of time; not all passport strategies require the same resolution.

If you’re looking closely, you’ll see the passport-local strategies both have return done(null, user); but the passport-jwt strategy has done (null, user);. See the difference? It’s minuscule, but having (or removing that return), is the difference between that user data being passed back from the middleware to the server or not.

And it halted my progress for a good few hours, before the kindness of Stack Overflow helped me work through the issue (it’s the first time I’ve actually had to ask SO for an answer I couldn’t find already, but it was well worth it). So be aware of when to return or not.

Ok, so I’ve walked through the two Passport strategies and the three methods I’m using for middleware authentication in my program, now on to after the authentication on the server.

The `register.js` File

The next step is how to implement these newly minted methods inside of the various routes. Here’s the registerUser route.

registerUser.js

import passport from 'passport';
import User from '../sequelize';

module.exports = app => {
  app.post('/registerUser', (req, res, next) => {
    passport.authenticate('register', (err, user, info) => {
      if (err) {
        console.error(err);
      }
      if (info !== undefined) {
        console.error(info.message);
        res.status(403).send(info.message);
      } else {
        req.logIn(user, error => {
          const data = {
            first_name: req.body.first_name,
            last_name: req.body.last_name,
            email: req.body.email,
            username: user.username,
          };
          User.findOne({
            where: {
              username: data.username,
            },
          }).then(user => {
            user
              .update({
                first_name: data.first_name,
                last_name: data.last_name,
                email: data.email,
              })
              .then(() => {
                console.log('user created in db');
                res.status(200).send({ message: 'user created' });
              });
          });
        });
      }
    })(req, res, next);
  });
};

As you can see, this implementation of Passport looks a little different than most examples out there, and it’s because I’m using the custom callback version of Passport, which requires a closure call. I’m using this so that the error message that’s passed back if Passport’s authentication fails can be sent from the server to the client instead of the obscure 401 Unauthorized.

Because this is being implemented as a callback (which is why you see (req, res, next) not once, but twice in these scripts; it gives me access to the (err, user, info) from the Passport middleware.

The info is what I’m interested in — that’s the message being sent back, so if the info is anything besides null, that means the authentication failed and I can then send the message to the client to let them know why.

Passport.js Gotcha #3: Don't Forget `req.logIn()` in Your Custom Callback

Now, I’m to the third Passport gotcha, in order to make the closures and custom callbacks work with passport-local, the little method req.logIn() must be called before the user data is handled if it comes back successfully from the middleware.

This is documented in the Passport documentation, but it’s importance is not stressed as much as I would like (and I missed it the first few times trying to get custom callbacks to work), which is why I highlight it now.

It must happen, and once that’s done, I’m able to take the extra inputs from the client side for registering a user, find that same user created in the database during the passport-local register call, and update the user file with the extra info. I could have passed this extra data through to the middleware as well, but I want Passport to only handle authentication, not user creation as well. Modularization, remember.

Plus, if the authentication were to be split out into a separate service with a separate database of just encrypted usernames and passwords, this would make it easier to do so, then use the username or ID to find and update the corresponding user record in this registration service.

Once all that’s been successfully taken care of, a 200 HTTP status and success message are sent back to the client.

The `loginUser.js` File

Seeing the loginUser route now should make more sense.

loginUser.js

import jwt from 'jsonwebtoken';
import passport from 'passport';
import jwtSecret from '../config/jwtConfig';
import User from '../sequelize';

module.exports = app => {
  app.post('/loginUser', (req, res, next) => {
    passport.authenticate('login', (err, users, info) => {
      if (err) {
        console.error(`error ${err}`);
      }
      if (info !== undefined) {
        console.error(info.message);
        if (info.message === 'bad username') {
          res.status(401).send(info.message);
        } else {
          res.status(403).send(info.message);
        }
      } else {
        req.logIn(users, () => {
          User.findOne({
            where: {
              username: req.body.username,
            },
          }).then(user => {
            const token = jwt.sign({ id: user.id }, jwtSecret.secret, {
              expiresIn: 60 * 60,
            });
            res.status(200).send({
              auth: true,
              token,
              message: 'user found & logged in',
            });
          });
        });
      }
    })(req, res, next);
  });
};

The same style of custom callbacks and closures are used, the biggest difference is that once the user is successfully verified and located in the database, the JWT token is generated using the jwt.sign(); function, which sets the username as the ID passed in the JWT, and encrypted by the secret I set earlier.

Once again, if all this works successfully, a 200 HTTP status is sent with a boolean I named auth set to true for the client side, the newly generated token and a short login success message.

The `findUsers.js` File

And last, but not least, here’s the findUsers route. I used the same passport-jwt authentication for the updateUser and deleteUser routes, so I’ll just show this one as an example.

findUsers.js

import passport from 'passport';
import User from '../sequelize';

module.exports = (app) => {
  app.get('/findUser', (req, res, next) => {
    passport.authenticate('jwt', { session: false }, (err, user, info) => {
      if (err) {
        console.log(err);
      }
      if (info !== undefined) {
        console.log(info.message);
        res.status(401).send(info.message);
      } else if (user.username === req.query.username) {
        User.findOne({
          where: {
            username: req.query.username,
          },
        }).then((userInfo) => {
          if (userInfo != null) {
            console.log('user found in db from findUsers');
            res.status(200).send({
              auth: true,
              first_name: userInfo.first_name,
              last_name: userInfo.last_name,
              email: userInfo.email,
              username: userInfo.username,
              password: userInfo.password,
              message: 'user found in db',
            });
          } else {
            console.error('no user exists in db with that username');
            res.status(401).send('no user exists in db with that username');
          }
        });
      } else {
        console.error('jwt id and username do not match');
        res.status(403).send('username and jwt token do not match');
      }
    })(req, res, next);
  });
};

I used custom callbacks for all my Passport authenticated routes (I appreciate good, clear error handling), so this code style should look routine by now.

This time, when passport.authenticate() is called, I implement the JWT strategy defined in the passport.js file. The same (err, user, info) gets passed back from the middleware, but there’s no call from req.logIn() this time.

Instead, I just return the user object found during the authentication and pass all the required fields to the client along with the auth boolean and success message. The JWT token is still stored in local storage on the client side so there’s no need to regenerate or pass it to the client again as well. Which brings me to my final Passport gotcha.

Passport.js Gotcha #4: Passing Authorization Headers Correctly

This one’s not exactly a Passport-specific gotcha, but it is yet another thing that tripped me up.

As I said, I pieced together my (now much more whole) understanding of JWT and Passport from a bunch of other tutorials and documentation, and one of those tutorials led me astray. It had me passing the JWT back to the client in something besides true authorization headers, which is the way that I chose to have my passport-jwt strategy extract the JWT payload from the client request.

For this reason, I wanted to touch briefly on one piece of the frontend code, where I pass back the token in the proper format, so you don’t have to waste the time debugging like I did.

Profile.js

 async componentDidMount() {
    const accessString = localStorage.getItem('JWT');
    const {
      match: {
        params: { username },
      },
    } = this.props;

    if (accessString == null) {
      this.setState({
        isLoading: false,
        error: true,
      });
    } else {
      try {
        const response = await axios.get('http://localhost:3003/findUser', {
          params: {
            username,
          },
          headers: { Authorization: `JWT ${accessString}` },
        });
        this.setState({
          first_name: response.data.first_name,
          last_name: response.data.last_name,
          email: response.data.email,
          username: response.data.username,
          password: response.data.password,
          isLoading: false,
          error: false,
        });
      } catch (error) {
        console.error(error.response.data);
        this.setState({
          error: true,
        });
      }
    }
  }

I’m using the popular promise-based HTTP client Axios to make my server calls, but what you really need to notice is the headers section.

Before the call to the server happens, I extract the JWT from local storage using localStorage.getItem('JWT') (the key I set with the JWT value when I passed the token forward from the loginUser() route on the server side), and then I set it using headers: { Authorization: JWT ${accessString} } and a little ES6 string interpolation. That way, if my authorization headers had more than one value to parse out, it would still be easy for passport-jwt to extract the correct JWT payload info by finding the string associated with "JWT".

And that’s it, I just wanted to cover it because it blocked me for a while and only through API testing with Insomnia, could I confirm my JWT token was actually working, and suss out that I was passing authorization headers from the client side incorrectly.

Conclusion

I know I wrote a lot (and provided a bunch of code snippets and gists), but authentication is not a simple concept. Nor is it truly possible to really make an app so secure no one will ever be able to hack it. But if you take precautions like JWT and Passport.js authentication middleware, your Node.js app should have a better chance to withstand people with malicious intent.

Just be sure to look out for the gotchas I highlighted around Passport — and you’ll be on your way to a more secure site.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading, I hope this proves helpful and gives you a better understanding of implementing Passport authentication and using JSON web tokens.

References & Further Resources

Passport.js website
JSON Web Tokens website
MERN App with Passport & JWT repo
Gists of Passport implementation with routes
Axios HTTP library

Postman vs. Insomnia: Comparing the API Testing Tools

Sat, 01 Sep 2018 00:00:00 GMT

</figure>

Introduction

Are you familiar with API endpoint testing? If you’re not, let me give you a quick definition before we go any further.

API testing is a type of software testing that involves testing application programming interfaces (APIs) directly and as part of integration testing to determine if they meet expectations for functionality, reliability, performance, and security.

— Wikipedia, API testing

API endpoint testing is one of the most important things we do as web developers. If the routes to reach our data are incorrect, the required parameters are not included, the authorization is missing, or a host of other things aren’t correctly hooked up, your app doesn’t work. If your app doesn’t work, your customers can’t use it. If your customers can’t use it, it’s going to be a really bad day for you, your team, your boss, and so on and so forth.

Which is why we have things like API testing tools. These tools allow us, as web developers, to test that when we pass a specific set of data (or not) to our application it will return the response we expect. It’s a pretty simple idea, but it makes our lives (and development cycles) so much easier.

NOTE: This will NOT be a blog post about how to use Postman or Insomnia (there’s plenty of those already, including the documentation on their own websites). This will be a comparison of the features the two offer, to help you make a better decision of which to use to suit your dev team’s needs.

API Testing Tools Benefits

</figure>

Here’s some of the biggest benefits to using an API testing tool, as briefly mentioned above.

Quicker setup and testing of endpoints (you don’t have to go through the whole user flow to set up data and then execute the commands to change the data),
Less command line interaction,
Multiple ways to set up authorization depending on which type you’re using,
Nicer formatting of code being sent and received,
Easier performance and reliability testing.

The Contenders: Postman & Insomnia

There’s two popular API testing tools I’ll be talking about today. Postman, a suite of tools which bills its ecosystem as:

“the Only Complete API Development Environment” — Postman

And Insomnia, a REST client which claims you can

“debug APIs like a human, not a robot” — Insomnia

So without further ado, let’s start by seeing what functionalities they share.

What They Both Offer

Insomnia and Postman both share a lot of benefits that are extremely helpful to developers on front-end teams, back-end teams and full-stack teams.

Among other things I’m sure I’m overlooking, both boast:

A free version of their software for users (upgradeable to more feature-rich paid versions at any point in time),
Allowing for multiple workspaces (e.g. development, acceptance, prod, etc.),
The ability to set environment and static local variables that get updated with each call to an endpoint (sometimes referred to as request chaining),
Allowing for imports and exports of test data for easy sharing between team members,
The ability to save calls and organize them into folders or collections,
GraphQL integration and testing options,
Multiple ways to configure authorizations (OAuth, bearer tokens, Basic, HAWK, etc.) and generate / manage cookies,
And both have light and dark color schemes, (I know it’s not a big deal, but it’s worth noting. I really prefer developing in a dark IDE to a stark white one).

That’s a good bit of stuff (and I’ll tell you, I’ve used the free versions of both Postman and Insomnia for enterprise level application development, and my teams and I have not felt the need to upgrade to a paid version of either, for what it’s worth).

Ok, now on to some of the unique offerings of both tools.

What’s Unique About Postman

</figure>

Postman appears to have a much more mature set of products and features when you start to really dive into its documentation.

Here’s some of the things that set Postman apart from other API testing tools.

API documentation (Postman will generate and host browser-based API documentation in real-time),
Collection runs (running a group of requests as a series against a corresponding environment — this is super helpful for automated testing, and I love the name of their collection runner: Newman — get it? Newman was the postman on Seinfeld… I love the sense of humor there),
Monitoring where Postman will periodically run a collection to check for its performance and response,
Tests written in plain JavaScript that will verify the response object, time, etc. from the endpoint,
And mock servers for teams to simulate each endpoint and its corresponding response in a Postman Collection. Developers can view potential responses, without spinning up a back end, and team members can be aligned on expectations during the earliest phases of API development.

Pretty impressive, right?

What’s Unique About Insomnia

</figure>

Insomnia, in its defense, has some cool things it offers that Postman does not.

It's open source.
Template tags (similar to environment variables but they perform operations on things like strings, timestamps, etc.),
The ability to create new plugins for Insomnia and its user community,
Client certificate assignments to workspaces and SSL validation (or the disabling of it),
Code snippet generation in 12 different languages (handy if you want to CURL the command from the command line or drop it in to your actual code base),
A very thorough documentation area where things like instructions, code snippets and even test data can be added to specific calls or collections,
Viewing response beyond JSON and XML (with Insomnia you can see HTML pages, images, SVGs, audio files and even PDF Documents).

Also, useful, handy things to have access to when developing APIs.

Conclusion

Overall, I have to say Postman comes out as the current leader in terms of API testing tools, in my mind. It’s more mature and full featured than Insomnia, offering a lot of really excellent benefits to developers, even in its free tier.

For my teams, who practice fully automated CI/CD testing and deployments, having a test collection runner like Postman offers is a big selling point, and if you’re fully dedicated to supporting an API, having generated, real-time documentation could also be a very nice to have.

That being said, if Insomnia incorporates a few of the benefits of Postman as it continues to develop and improve, there’s no reason it couldn’t continue to be a very strong competitor in the API testing world.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading, I hope this proves helpful and gives you a better understanding of a couple of the major options out there for testing.

References & Further Resources

Using Docker & Docker Compose to Improve Your Full Stack App Development

Fri, 17 Aug 2018 00:00:00 GMT

</figure>

Introduction

I don’t know about you, but for me, oftentimes the hardest part of a new project is getting all the necessary pieces up and running. The databases, the backend servers, and of course, the frontend UI — it’s a lot to keep track of, manage dependencies of, monitor the health of, prevent port collisions of, and make sure each of the components can connect to the others to make the app work from start to finish.

And don’t get me started on if you’re developing with a team of people, and you’re all connected to the same database. Let’s just say, it can be a recipe for a lot of headaches and the oft heard phrase “Did someone blow away the database again? The <insert data of your choice> I was testing with is gone!" It’s not pretty, and it’s definitely not fun when the data gets out of sync or flat out deleted.

Today, I'll show you how containerized development with Docker can make everyone's local development experience better.

The Solution? Containerized Development Environments

<figcaption>I ❤️ the Docker mascot, Moby Dock. He’s just so darn cute.</figcaption>

</figure>

I’m going to radically improve how you develop — at least locally, and possibly in your lower life cycles and production environments as well. How? With the help of our trusty docker-compose.yml file. If you’re not familiar with Docker, I recommend you check out two of my previous blog posts covering the basics of Docker and one of Docker’s most useful features, Docker Compose.

If you are familiar with Docker already, please read on. You know Docker, you’re aware of the containerized nature of it, and you may even have used the power of containers or whole containerized ecosystems courtesy of Docker Compose.

But have you thought about how it might make your localized application development easier? Think about it: using a docker-compose.yml to control your dev environment solves the problems I mentioned above.

All the services listed in the docker-compose.yml can be started up together with just one command,
There’s no chance of port collisions (at least in the internal Docker environment) even if applications start up on the same port,
Each of the services is aware of, and able to connect to, the other services with out issue,
There’s less of a chance of the “It works on my machine” syndrome, since every container is using the exact same image with the exact same dependencies,
And best of all each individual Docker environment can have its own databases, that no one else can access (and subsequently ruin the data of).

Plus, it’s super simple to do. Intrigued yet?

How To Dockerize Applications

I’ll be demonstrating the ease of "dockerizing" an application with a full stack JavaScript application. This is a MERN application, except replace the MongoDB with a MySQL database.

If you want to see my full application, the code is available here in Github.

Here’s a high level look at the file structure of this particular application. Since I’m demonstrating with a relatively small user registration app, I’m keeping the server and client in the same repo, but it would be very easy to break them up into multiple separate projects and knit them together with the help of docker-compose.ymls.

Side note: If these were to be two (or more) separate services, I would need to keep a docker-compose.yml in both repos so each could be spun up for local development.

The service being developed would build a new image each time docker-compose up is invoked, and it would have access to the necessary other service’s Docker image stored in Docker Hub (or wherever you choose to store images) with the tag of the version I wanted in the docker-compose.yml.

For production, there’d also be a third docker-compose-prod.yml which would only use images that were tested and approved. But that’s outside the scope of this article.

Back to the project file structure.

App File Structure

root/
├── api/ 
├── client/ 
├── docker/ 
├── docker-compose.yml 
├── Dockerfile

Obviously, there’s plenty of files contained within each one of these directories, but for simplicity’s sake I’m just showing the main file structure.

Even though both the client/ and api/ folder are contained within the same repo, I built them with micro services and modularity in mind. If one piece becomes a bottleneck and needs a second instance, or the app grows too large and needs to be split apart, it’s possible to do so without too much refactoring. To achieve this modularity, both my API and client applications have their own package.json files with the dependencies each app needs to run.

The nice thing is, since this is currently one application and both apps are JavaScript, I can have one Dockerfile that works for both.

The Dockerfile

Here’s what the Dockerfile looks like:

// download a base version of node from Docker Hub
FROM node:9

// create the working directory for the application called /app that will be the root
WORKDIR /app

// npm install the dependencies and run the start script from each package.json
CMD ls -ltr && npm install && npm start

That’s all there is to it for this Dockerfile: just those three commands. The docker-compose.yml has a little bit more to it, but it’s still easy to follow, when it’s broken down.

The Docker-Compose.yml

version: '3.1'

services:
  client:
    build: .
    volumes:
      - "./client:/app"
    ports:
      - "3031:3000"
    depends_on:
      - api

  api:
    build: .
    volumes:
      - "./api:/app"
    ports:
      - "3003:3000"
    depends_on:
      - db

  db:
    image: mysql:5.7
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: example
      MYSQL_DATABASE: users
      MYSQL_USER: test
      MYSQL_PASSWORD: test1234
    ports:
      - "3307:3306"
    volumes:
      - ./docker/data/db:/var/lib/mysql

I would like to note there’s a newer version of MySQL available on Docker Hub, but I couldn’t get it to work properly with Docker Compose, so I would recommend going with the mysql:5.7 version until further notice.

In the code snippet above, you can see my two services: the api and client , as well as the database (the first mention I’ve made of it thus far, and one of my favorite things about Docker Compose).

To explain what’s happening, both the client and API services are using the same Dockerfile, which is located at the root of the project (earning them both the build: . build path. Likewise, each folder is mounted inside of the working directory we specified in the Dockerfile with the volumes: ./<client or api directory>:/app. I exposed a port for each service to make debugging of individual services easier, but it’d be perfectly fine to only expose a port to the application through the UI. And finally, the depends_on makes each piece of the app wait until all the parts have started up.

The client depends on the api starting, the api depends on the database starting, and once all the credentials are supplied to the database and the image is pulled from Docker Hub, it can start up. I opened a port on the database so I could connect Sequel Pro to my database and see the user objects as they’re made and updated, as well. Once again, this makes debugging as I develop the application easier.

The very last line of the database about volumes, is a special line that deserves attention. This is how data is persisted for the application. It’s only persisted locally to the machine the Docker ecosystem is running on, but that’s usually all you need for development. This way, if you’re using Flyway or Liquibase or another SQL runner to create the tables and load data into them and you then change that data, to test the app’s functionality, the changes can be saved so that when you restart the app, the data is the way you left it. It’s really awesome.

Ok, so the Dockerfile has been covered, the docker-compose.yml has been explained, and the database image being pulled from Docker Hub has been noted. We’re about ready to roll.

Start the App With One Line

Now it’s time to start this application up. If it’s your first time developing this application locally, type docker-compose build in the command line. This will build your two images for the client and API applications — the MySQL database comes as an image straight from Docker Hub, so there’s no need to build that image locally. Here’s what you’ll see in the terminal.

<figcaption>You can see the db is skipped and the API and the client are both built using the Dockerfile at the project’s root.</figcaption>

</figure>

Once the images are done building, type docker-compose up. You should see a message in the terminal of all the services starting up and then lots of code logging as each piece fires itself up and connects. And you should be good to go. That’s it. Up and running. You’re done. Commence development.

<figcaption>This is what you’ll see right after writing docker-compose up. Once all the services register done, they’ll try to start and connect and you should be good to start developing.</figcaption>

</figure>

Whenever you want to stop your app, you can just type docker-compose down into the terminal, and the services will gracefully shut down. And the data will be persisted locally, so when you type docker-compose up to start the applications back up, your data will still be there.

<figcaption>What you see when you stop the services with docker-compose down.</figcaption>

</figure>

Conclusion

Docker and Docker Compose can make web development a heck of a lot easier for you. You can create fully functional, isolated development environments complete with their own databases and data with very little effort on your part, speeding up development time and reducing or avoiding the problems that typically arise as projects are configured and built by teams. If you haven’t considered "dockerizing" your development process yet, I’d highly recommend looking into it.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading, I hope this proves helpful and makes your own local development easier for everyone on your development team.

References & Further Resources

Sequelize: Like Mongoose but for SQL

Sat, 04 Aug 2018 00:00:00 GMT

</figure>

Introduction

Everyone who's ever built a MEAN or MERN stack app is probably familiar with Mongoose, the helpful object data modeling library that makes working with a MongoDB much, much easier. And since this application stack is so popular, there's about a thousand tutorials (some much better than others) illustrating how to use MongoDB, Express, React (or Angular) and Node.js together.

Imagine my surprise though, when I decided to build a simple full stack app using MySQL (or SQLite or PostgreSQL or really any type of SQL database), and found almost nothing that made much sense on how to do so using Express and Node. Really? Nothing recent? Nothing with a deliberate, modularized architecture? No one seemed to have a good way of doing this - nothing short of actual SQL queries written out in the server.js file to perform create, read, update, and delete (CRUD) operations on the database.

Then, I found Sequelize, and everything changed.

I wanted to share what I learned here, in hopes of saving some other people from the masses of outdated tutorials out there, unaware of how much easier Sequelize can make implementing a SQL database into their own Node apps.

What is Sequelize?

Sequelize's home page best sums up what exactly it is:

Sequelize is a promise-based ORM for Node.js v4 and up. It supports the dialects PostgreSQL, MySQL, SQLite and MSSQL and features solid transaction support, relations, read replication and more. - Sequelize

It is the easiest and most straightforward tool I've found for working with databases in SQL inside of a JavaScript application. There are other options that offer similar functionality. Knex.js is one that comes to mind, but after comparing their GitHub stats, I chose to move forward with Sequelize.

Why Sequelize in particular?

Here's some stats that helped me make my decision:

Sequelize has over 8,000 commits (Knex has just under 2,000),
It has 375 releases (Knex has 119),
694 contributors (Knex has 237),
It recently released the beta version 5 (Knex is on v0.15),
And it has a CLI, TypeScript and GraphQL integration (Knex does not),
Plus 883,000 downloads per month from NPM (Knex has just under 600,000 per month).

In almost every way (by the numbers), Sequelize is superior to Knex, so I decided to go with what seems to be a very stable, well documented, long maintained and frequently updated library.

I will say that Knex offers native Oracle and Amazon Redshift support, but since I wasn't looking to use an Oracle DB, this wasn't a major selling point for me.

Sequelize compared to Mongoose

So, when you use Sequelize, you get a lot of the same types of benefits Mongoose offers to MongoDB users plus some bonuses:

The ability to connect with very little code changes to multiple different types of SQL databases,
Object models (similar to Mongoose schemas) which are typed, mapped to tables and can be required,
Basic and more advanced querying options like findAll/findOne, destroy, update, etc. plus things like pagination, nested objects, and associations,
Connection pooling,
Raw queries,
And more I don't have time to cover.

The documentation is very thorough and shows just how much Sequelize can offer over having to manage it all yourself.

I'd also like to note that by using Sequelize you can save yourself the trouble of having to rewrite SQL queries to fit the slight variations in database dialects - your find and create statements need no modifications to work regardless of which type of DB you're connecting to. I don't know about you, but I'm a big fan of less effort on my end.

Sequelize also uses promises via Bluebird to control its async control flow. All this means is that when you query the database via Sequelize, it will return a promise like this one.

User.findOne().then(user => {   
    console.log(user.get('firstName')); 
});

// returns user object's first name, like 'John'

It also means that Sequelize works well with the ES6 async/await functionality like so.

user = await User.findOne();  

console.log(user.get('firstName'));

// also returns user object's first name, like 'John'

How to implement Sequelize in Express.js

Great, so now you can see some of the reasons to use Sequelize, how do you use it in your project?

This is always my biggest question whenever I see samples or tutorials. How do I use it? And how do I organize my files and my program structure so it's easy to swap pieces of functionality in and out with minimal modifications on my part?

Here's the source file structure I came up with (including just server (API) folders). It's pretty simple because the project I was trying to do wasn't particularly complex. But it offers enough organization to make adding CRUD functionality through the routes folder easy and defining one or more database models is as well thanks to the models folder.

root/
├── api/ 
| ├── server.js 
| ├── sequelize.js 
| ├── package.json
| ├── models/ 
| | ├── user.js
| ├── routes/ 
| | ├── deleteUser.js 
| | ├── findUser.js 
| | ├── loginUser.js 
| | ├── registerUser.js
| | ├── updateUser.js
| ├── data/ (optional depending on storage choice - like SQLite) 
| | ├── db/ 
| | | ├── storage.sqlite
| ├── node-modules/

So, now that you see the file structure, let me show you what your files will contain. I'll start with the user.js file because it gets imported into the sequlize.js and routes files.

Define the models

Below is the code for a user model.

user.js

module.exports = (sequelize, type) => {
  return sequelize.define('user', {
    id: {
      type: type.INTEGER,
      primaryKey: true,
      autoIncrement: true
    },
    first_name: type.STRING,
    last_name: type.STRING,
    email: type.STRING,
    username: {
      type: type.STRING,
      allowNull: false
    }, 
    password: {
      type: type: STRING,
      allowNull: false
    }
  })
};

Since I was building a user registration application, my user model file was very straightforward. Just an ID that's been set as the primary key and will auto increment, a first name, last name, email, username and password (both of which are required). It's pretty easy to follow.

Set up the database connection(s)

Next up is the sequelize.js file, this is where I kept the majority of my database setup to make it easy to switch out databases if you ever wanted to migrate from say a MySQL DB to a SQLite DB. Here's what a MySQL connection looked like:

sequelize.js

import Sequelize from 'sequelize';
import UserModel from './modes/user';

const sequelize = new Sequelize('dbname', 'admin', 'password', {
  host: 'db',
  dialect: 'mysql'
});

const User = UserModel(sequelize, Sequelize);

sequelize.sync()
  .then(() => {
    console.log('Users db and user table have been created');
  });

module.exports = User;

As you can see, first, I initiate the connection to the MySQL database with the sequelize variable, giving it a database name, username, password, and then specifying the host and dialect. That's all there is to it.

Then I added the user model defined earlier, and the sequelize.sync() function syncs all defined models to the database. In this way, it creates the database and tables based on the information you've given it and models you've passed to it.

Finally, I export the User constant that I defined earlier, for use in other parts of the project, like the routes.

Here's another example of a SQLite connection so you can see the differences:

sqliteSequelize.js

import Sequelize from 'sequelize';
import UserModel from './modes/user';

const sequelizeSqlite = new Sequelize('dbname', 'admin', 'password', {
  host: 'sqlite',
  dialect: 'data/db/storage.sqlite'
});

const UserSqlite = UserModel(sequelizeSqlite, Sequelize);

sequelizeSqlite.sync()
  .then(() => {
    console.log('Users db and user table have been created with Sqlite');
  });

module.exports = UserSqlite;

A simple connection to a SQLite database with storage persisted within the project itself.

Besides the slight changes to the database set up, everything else in the file, stays the same.

Define the routes

Ok, I'm almost done with setting this up.

To keep the code modular (and make it easier for me to add or subtract functionality from the application), I added each piece of the CRUD functionality to the database as a separate file. You could add them all in the same file and keep it divided up by model, or whatever is easiest for you to keep things organized.

Here's an example of my user registration route (the create functionality). I added in password hashing through the use of bcrypt, but that's not important right now.

registerUser.js

import User from '../sequelize';
import bcrypt from 'bcrypt';

const BCRYPT_SALT_ROUNDS = 12;

module.exports = (app) => {
  app.post('/registerUser', (req, res) => {
    const data = {
      fist_name: req.body.first_name,
      last_name: req.body.last_name,
      email: req.body.email,
      username: req.body.username,
      password: req.body.password
    };

    if(data.password === '' || data.username === ''){
      res.json('username and password required');
    }
    User.findOne({
      where: {
        username: data.username
      }
    })
    .then(user => {
      if (user !== null) {
        console.log('username already taken');
        res.json('username already taken');
      } else {
        bcrypt.hash(data.password, BCRYPT_SALT_ROUNDS)
        .then((hashedPassword) => {
          User.create({
            first_name: data.first_name,
            last_name: data.last_name,
            email: data.email,
            username: data.username,
            password: hashedPassword
          })
        })
        .then(() => {
          console.log('user created');
          res.json('user created');
        })
      }
    })
    .catch(err => {
      console.log('problem communicating with db');
      res.status(500).json(err);
    })
  })
};

As you can see from the code above, the User model is imported from the sequelize.js file, the user input data is passed from the client side of the app through the req.body.xyz fields, and then I do a number of checks to the database with the help of Sequelize before adding a new user.

The first check is when I check to make sure the user supplied both a username and password (as required by the model), if they're not both there, the server sends back the error message 'username and password required'.

After that check, I query the database to see if that username is already taken, if it is, once again the server sends 'username is already taken' back to the client.

If it passes that check, the password is hashed and the user info is saved to the database. Once it's successfully saved it sends back a success message to the client 'user created'.

And the whole creation operation is wrapped so in case there's a problem communicating with the server (it's down, there's network issues, etc.), it can still send a message to the client letting it know there was a problem.

Add the routes to the server

Now I'm ready to add these routes to the server. This is easiest part of all.

server.js

import express from 'express';
import Cors from 'cors';
import bodyParser from 'body-parser';
import logger from 'morgan';

const app = express();

app.use(Cors());
app.use(bodyParser.urlencoded({ extended: true }));
app.use(bodyParser.json());
app.use(logger('dev'));

require('./routes/loginUser')(app);
require('./routes/registerUser')(app);
require('./routes/findUsers')(app);
require('./routes/deleteUser')(app);
require('./routes/updateUser')(app);

app.listen(API_PORT, () => console.log(`Listening on port ${API_PORT}`));

module.exports = app;

All you have to do to make this work, is require the routes in your server.js file like I did, and have them take in your Express application, and you're set. It doesn't get much simpler than that.

Now, you should be able to use Sequelize to your heart's content with any kind of SQL database in your Node.js, Express project.

If you'd like to see a full MERN stack example using Sequelize, you can see my repo here. From the master branch you can run docker-compose build, followed by docker-compose up, and from the sqlite feature branch you can start both the client and server files individually with npm start from the api/ and client/ folders.

Conclusion

Sequelize brings order and flexibility to using SQL databases in JavaScript applications. It reduces the need to write raw queries, manage connections, guard against harmful SQL attacks, and more. And with some fairly basic setup, you too, can use Sequelize in your own JS projects - and hopefully speed up your development time. I will definitely continue to use this tool going forward.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading, I hope this proves helpful in your future database-driven projects.

References & Further Resources

Jib: Getting Expert Docker Results Without Any Knowledge of Docker

Wed, 18 Jul 2018 00:00:00 GMT

</figure>

Introduction

Well have I got some news for you. If you’ve been following my latest blog posts, I’ve been talking a good deal about Docker, the amazing containerization platform making it easier than ever to develop and deploy web applications. The first post covered the Docker basics and the Docker CLI. The second , introduced Docker Compose, Docker’s more sophisticated, easier (in my opinion) to use, multi-service option.

Docker is a very powerful tool when used correctly, but it takes a minute to get your head around:

The commands,
The file structure,
The whole containerization concept, itself.

It’s not easy if you’ve not used it before, there’s a definite learning curve. But Google, ever the leader in technology, has just released a game changer…

Meet Jib

Jib, is a brand new, just released, Google Container Tools-based, Java plugin that will containerize your application.

Put simply:

Jib builds Docker and OCI images for your Java applications and is available as plugins for Maven and Gradle. — Jib, Github

What this means for us is:

No Dockerfile setup,
No customized scripting for Maven or Gradle,
Out of the box Docker image generation and publication.

Basically, no Docker complexity, but all of the Docker benefits.

How? A plugin, a very simple plugin in your project. I’ll show you just how simple below.

Implementing Jib in Gradle

There are both Maven and Gradle plugins for Jib. I’ll be using Gradle because that’s what I’m most familiar with.

We begin with the build.gradle file. First and foremost, make sure you’re working with Gradle 4.6 or later — this is required for Jib.

Next, the changes to the file. Inside of the build.gradle, in your plugins section, add this line.

plugins {
  id 'com.google.cloud.tools.jib' version '0.9.6'
}

Yes, it really is that easy to generate a Docker image now.

And that’s it. You’re done with the changes needed in your Java application.

Now, from the command line run the Gradle task ./gradlew jibDockerBuild. This will generate your local Docker image and give you its image name and version.

Ok, final step, the Docker run command in the terminal: docker run -p <host port: Docker app port> <image name: version>. And with this your application should start up.

I made a sample Java application and stored it in Github here, so you can see just how simple this really is.

I also left the Dockerfile in there, so you can see what used to be necessary to run Docker. Keep in mind, this project is only set up for building and running local images, if you want to push to Docker hub, see the Jib official documentation (and check back, I may do a follow up post on it).

For my project, you’d run the same ./gradlew jibDockerBuild, then this Docker run command: docker run -p 8080:8080 java-example:1.0.0-SNAPSHOT.

<figcaption>This is what you’ll see when you go to http://localhost:8080 in your browser.</figcaption>

</figure>

Before Jib, I needed an entirely separate Dockerfile instructing how to build my application to make magic like this happen.

Dockerfile

FROM openjdk:8-jdk-alpine

# create the directory for where Tomcat creates its working directories
VOLUME /tmp

# copy the project JAR file to the container renamed as 'app.jar'
COPY build/libs /app

# execute that JAR in the entry point below
# java -Djava.security.egd=file:/dev/./urandom -jar /app/java-example.jar
ENTRYPOINT ["java", "-Djava.security.egd=file:/dev/./urandom", "-jar", "/app/java-example.jar"]

Look at this. And this is a simple, simple Java application. These Dockerfiles can (and usually are) much bigger and more complex.

Conclusion

Thanks to Jib, Dockerfiles can be a thing of the past — at least, for your Java applications. Another bonus is that if you have a docker-compose.yml file in your project, that can still be used in tandem with Jib.

Personally, I can’t wait to start dropping this one-liner into all of my team’s Spring Boot applications. With it, we can get the power and flexibility of a Docker container with none of the setup and necessary knowledge about Docker.

Even if you’re not a Docker expert, with Jib you can still reap the benefits, and look like a rockstar in the process. Enjoy!

References & Further Resources

Docker 102: Docker-Compose

Sat, 07 Jul 2018 00:00:00 GMT

</figure>

Introduction

If you’re not familiar with Docker, the containerization platform, you may want to read my first blog, to get an overview of what it is and what happens under the hood before embarking on the quest to understand Docker-Compose.

If you do know about Docker, read on, and I’ll attempt to show you the power and benefits a single docker-compose.yml can offer over the original Dockerfile.

Which segues nicely to the first point I’ll cover: why use Docker Compose instead of the Docker CLI commands?

Why Docker Compose?

Docker Compose offers a number of benefits over the Docker CLI in all stages of development. Here, I’ll give you some details of where it can best be utilized in each stage.

Lower Life Cycles

For lower life cycle environments, Docker Compose is able to quickly, easily simulate production environments.

It can spin up all the microservices (and databases) that make an application run, and it can seamlessly connect them to each other inside of their isolated Docker environment with very little effort on the developers’ part. There’s no need to even specify different ports internally to avoid collisions — it’s extremely easy to get the whole system running in one fell swoop.

And then there’s testing. Docker excels at many types of application testing:

Destructive testing
Performance testing
Integration testing
End-to-end testing

Docker performs so well for testing because it creates its own isolated environment, spins up its own databases, sets up the data if directed to do so, and when it’s done running tests and manipulating the data, it can shut down and start back up again with the original data in the same pristine condition as before. How handy is that as your multiple devs are working on different features and submitting PRs only after there’s been a clean run of end to end tests?

Production

If you’re lucky enough to be able to use Docker containers in production, then a single docker-compose.yml file can deploy a complete system that runs exactly as it was tested in lower life cycle environments. This can happen because exactly the same steps done in development can be done in production with Docker Compose, it’s awesome and helps eliminate some of the trepidation deployments always cause that something might go wrong.

Docker Swarm Testing

Docker Compose makes Docker Swarm testing before deployment easier. Docker Swarm is beyond the scope of this post, so just know this is the way to test the swarm.

Finally, in my humble opinion, a docker-compose.yml file is easier to read, understand and implement than the Docker CLI. This is purely my opinion, though.

So now that I’ve given you a reason for wanting to use Docker Compose, the next question is: what does a docker-compose.yml look like? And how do I write one? Read on.

What is Docker Compose?

Essentially, what Docker Compose is, is a recipe card — it’s a recipe of services that make up an application and the docker-compose.yml dictates how the services are mixed together.

In more technical terms:

Docker-Compose defines a complete system where services can communicate with each other on an isolated network, interacting with external resources as defined by the user.

Docker Compose is built on top of the Docker Engine, the very same engine that runs the Dockerfiles and Docker CLI, but the most important aspect to keep in mind about Compose is to think of it like a recipe.

The image below illustrates it pretty well.

</figure>

This is what a Docker Compose environment looks like when running. Application A is a backend microservice that can freely communicate with App B and App C but that is all, it cannot communicate with the host in the outside world at all. This is because of how its docker-compose.yml is set up, I’ll show that next.

Application B is a database, it has been structured in the docker-compose.yml with an externally mounted volume volumes: — opt/app:/opt/appB/app that allows it to write a persistent data layer to the outside world. With this data layer in the application, even if the entire Docker environment is destroyed for some reason, when it’s brought back up, the database will see the data written to it before, and replace it so you can begin where you were before.

Finally, Application C is the user interface portion of the app. Also, due to the docker-compose.yml, this application can talk to the host machine on a port that’s been exposed: ports="8080:8080". This means, a user can open up http://localhost:8080 on their machine and access the running application.

It's is a pretty simple representation of the environment, but it can be that simple, or it can be much more complex depending on your needs. Now, let’s look at what a Docker Compose YAML file looks like and go through some of the nuances that can come with it.

How to Docker Compose?

The last big question is: how to write a docker-compose.yml, and it’s actually very easy and follows a standard formula.

Here is a template of what any docker-compose.yml will look like.

Sample Docker Compose Template

version: "2"

  services:
    <name_of_service>:
      build: <path_to_dockerfile>
      # OR
      image: <name_of_image>
      environment: 
        - "java_home:/user/bin/default"
        - "app_dir:/opt/app"
      ports:
        - "80:80"
        - "8000:8000"
        - "443:443"
      volumes:
        - ./:/opt/app

Every docker-compose file will start with a minimum of version: "2", if you’re doing a Docker Swarm file it will need version: "3", but for a single docker-compose.yml, you’ll need v2.

Next, you will list all your services (each app running in the docker-compose will have its own name, and you’ll need to provide either a build or an image from the Docker registry. You cannot have both.

A custom Docker image will be built based on the Dockerfile provided by the build path OR
Docker Hub will supply all the necessary commands and setup to make the image build and run successfully.

Underneath that, environment variables are set, ports are exposed, and external volumes are mounted. All these same things could be done with the Docker CLI, but this is a cleaner, easier to read file that doesn’t require remembering (and correctly typing) them all when executing commands in the terminal.

Here is what a sample docker-compose.yml could look like.

Sample Docker Compose Yaml

version: "2"

  services:
    appA:
      build: ./Dockerfile
      ports:
        - "80:80"
    appB:
      image: mongodb
      volumes:
        - ./data:<wherever>
    appC:
      build: ./Dockerfile
      instances: 2
      environment:
        - config: backend

This docker-compose.yml begins with version: "2" (as required for Docker Compose), and then we have three services listed below: appA, appB, appC. Since appA is the UI that is built from a custom Dockerfile and needs to be exposed to the host machine, it has a build: path to said Dockerfile and a ports: that maps the internally running port of the UI, port 80 to the same port on the host machine, also 80.

appB is a Mongo database, so it can be pulled down from the Docker registry when it’s time to start up the application. There’s no build to be done ahead of time, nothing but declaring the image, and for data persistence adding a volumes: specification of where database data can be persisted to the host machine and it’s good to go.

And appC is also a custom Docker image, so it has a build: path to its own Dockerfile, an environment: variable specifying a configuration file for the backend, and not one but two instances:.

This application in particular can have two instances because it is not being exposed in any way to the host machine. If a port was being opened (or it was some type of database), it would only be able to have one instance to avoid port collisions on the host or to avoid writing to two separate databases within the Docker environment.

But since this is purely a backend service, it can have as many instances as necessary (good for load balancing heavy volumes of traffic), and all of them can run as separate containers within the Docker environment and communicate with each other as needed.

I recommend reading the Docker Compose documentation for more examples and descriptions of all the things you can do with within a docker-compose.yml.

Ok, the docker-compose.yml has been written, let’s look at the Docker Compose commands needed to get this system up and running. Don’t worry, they’re not very complicated.

Docker Compose Commands

Before I go any further, I must issue the following disclaimer.

DISCLAIMER: For the purposes of this blog post, it will be assumed all docker-compose commands will be executed in the same directory as the docker-compose.yml. Yes, it is possible to execute the commands from different levels of the directory, but that is outside of the scope of this blog.

Now that that’s been made clear, we can continue. There’s just a handful of Docker commands you’ll need to type in to the terminal to get these magical files to do their thing. And after I’ve described them, I’ve provided another handy graphic that illustrates the three different phases a Docker Compose environment can exist in.

docker-compose ps — lists all the services in a network. This is especially helpful when troubleshooting a service as it will give you the container ID and you can then run docker -it exec <ID> bash to enter the container and debug as needed.
docker-compose build — generates any needed images from custom Dockerfiles. It will not pull images from the Docker hub, only generate custom images.
docker-compose up — brings up the network for the services to run in
docker-compose stop — stops the network and saves the state of all the services
docker-compose start — restarts the services and brings them back up with the state they had when they were stopped
docker-compose down — burns the entire Docker network with fire. The network and all the services contained within are totally destroyed.

</figure>

Here’s the various states the Docker Compose environment exists in, depending on what commands you’ve run.

docker-compose build and docker-compose down mean the Docker environment is not running, and the network does not exist.

docker-compose up and docker-compose start means the Docker environment is running, and the network does exist.

docker-compose stop means the Docker environment is not running, but the network still does exist.

That’s it. All the commands you’ll need with your docker-compose.yml to build and run all your connected microservices.

Docker Compose Examples

If you’d like to see some examples of how Docker Compose can be structured, you can download a repo I put together here.

Each sample has a README.md that illustrates how to run it, and it illustrates the flexibility afforded with Docker Compose.

Example 1: Node.js and Java - Dockerfiles

The first example features one Node.js and one Spring Boot Java application, both of which have Dockerfiles but no docker-compose.yml, and it has basic Docker CLI commands to run them on ports 3003 and 3004, respectively.

This is to show that the very same instructions the docker-compose.yml contains can be done from the command line, but it takes more effort on the user’s part to get them running. Also, neither service is aware of the other one, the way they would when connected by a docker-compose.yml.

<figcaption>Node.js Docker example: running on port 3003</figcaption>

</figure>

<figcaption>Java Spring Boot Docker example: running on port 3004 and displaying its hostname </figcaption>

</figure>

Example 2: Node.js and Java - Docker Compose

The second example deploys the same two applications, but uses a single docker-compose.yml file to do so. This time, the Java app runs on 3001 and the Node app runs on 3002.

If you wanted to, you could have example 1 and example 2 all running at the same time since none of them share host port numbers.

<figcaption>Node.js docker-compose example: running on port 3002.</figcaption>

</figure>

<figcaption>Java Spring Boot docker-compose example: running on port 3001 and displaying a different hostname.</figcaption>

</figure>

Example 3: Node.js and Java - Nginx

Example 3 takes things up a notch by using a simple Nginx frontend to route requests through to either the Java or the Node service running in its network.

Nginx is outside the scope of this blog, but for our purposes it’s working as a proxy and depending on what route is passed to it, it will either serve up one app or the other.

This means there’s only one port exposed to the outside world (80 is the default port for Nginx, so that’s what I’m using), but both applications inside the Docker environment can be accessed, no ports for those services necessary. See the screenshots below to help clarify it a bit.

<figcaption>Look closely at the URL, it’s just http://localhost/node/, and it routes to the Node.js example.</figcaption>

</figure>

<figcaption>Likewise, the same port with the /java/ route, routes to the Java example.</figcaption>

</figure>

Example 4: Node.js and Java - Dockerfiles and Docker Compose

This is my favorite example by far, because it really helps illustrate the power of Docker Compose.

At the root level of the example directory, this particular docker-compose.yml accesses all the Java and Node.js examples at one time, builds their images from their individual Dockerfiles and spins all six of them up plus two more applications bearing image tags of latest. All of these containers are accessed through a single Nginx proxy as well.

It’s really, really cool and an example of how you could use versioning for blue-green production deployments, canary deployments, or even as a way to support multiple versions of the same application if different outside teams are dependent on a certain version for functionality.

Here’s just a couple examples of the versioned routes — pay attention to the URLs.

<figcaption>Example 3 of the app, served up on the v3 route.</figcaption>

</figure>

<figcaption>Example 1 on the v1 route.</figcaption>

</figure>

<figcaption>And example 1 again, but this time on the latest route tag. Nice.</figcaption>

</figure>

Conclusion

And that is Docker Compose in brief. Imagine writing a "recipe" of all the services that make up a single application, databases included, and Docker handles the rest.

Ports won't collide, container services can talk to one another, and it's all written down in an easy-to-read file that can be deployed in upper and lower lifecycles. It's really powerful.

Thanks for reading, I hope you’ll have the chance to use Docker Compose in your own development like I do.

References & Further Resources

Docker 101: Fundamentals & the Dockerfile

Sat, 23 Jun 2018 00:00:00 GMT

</figure>

Introduction

You, like me, may have heard of Docker before. You may have co-workers or developer friends who rave about it, who “dockerize” everything they can, and who are at an utter loss for words when it comes to explaining Docker, and all its glory, in terms that are understandable to the average person. Today, I will attempt to demystify Docker for you.

This will actually be a two-part series on Docker, because Docker needs, at least, two separate posts for its two best known products:

Docker (and the Dockerfile),
Docker Compose

So let’s begin at the beginning and I’ll explain what Docker actually is, in terms even I understand.

What is Docker?

The Docker website itself simply describes Docker as:

“the world’s leading software containerization platform” — Docker, Docker Overview

That really clears it up, right? Umm…yeah, not really. A much better description of Docker is found on OpenSource.com.

Docker is a tool designed to make it easier to create, deploy, and run applications by using containers. Containers allow a developer to package up an application with all of the parts it needs, such as libraries and other dependencies, and ship it all out as one package. — OpenSource.com, What is Docker?

What Docker really does is separate the application code from infrastructure requirements and needs.

It does this by running each application in an isolated environment called a ‘container.’

This means developers can concentrate on the actual code to run in the Docker container without worrying about the system it will ultimately run on, and devops can focus on ensuring the right programs are installed in the Docker container, and reduce the number of systems needed and complexity of maintaining said systems after deployment.

Which is a perfect segue to the next point: why Docker?

Why Docker?

I can’t tell you how many times I’ve heard a developer (myself included) say, “It works on my machine, I don’t know why it won’t work on yours.” — any developer, ever

This is what Docker is designed to prevent — the inevitable confusion that comes when a dev’s been working on their local machine on a project for days (or weeks), and as soon as it’s deployed to a new lifecycle, the app won’t run. Most likely because there’s a host of installed dependencies that are necessary to run the application but they aren’t saved in the package.json or the build.gradle or specified in the manifest.yml.

Every single Docker container begins as a pure vanilla Linux machine that knows nothing.

<figcaption>The Docker container’s just like Jon Snow when it spins up: it knows nothing.</figcaption>

</figure>

Then, we tell the container everything it needs to know — all the dependencies it needs to download and install in order to run the application. This process is done with a Dockerfile, but I’m getting ahead of myself.

For this section, suffice it to say, Docker takes away the guesswork (and hours spent debugging) from deploying applications because it always starts as a fresh, isolated machine and the very same dependencies get added to it. Every. Single. Time.

No environments that have different versions of dependencies installed. No environments missing dependencies entirely. None of that nonsense with Docker.

Docker Tools

Before I dive into the Dockerfile, I’ll give a quick run down of Docker’s suite of tools.

</figure>

Docker has four main tools that it provides to accomplish tasks:

The Docker Engine is Docker’s

“powerful open source containerization technology combined with a work flow for building and containerizing your applications.” — Docker, About Docker Engine

It’s what builds and executes Docker images from either a single Dockerfile or docker-compose.yml. When someone uses a docker command through the Docker CLI, it talks to this engine to do what needs to be done.

Docker Compose

“is a tool for defining and running multi-container Docker applications” — Docker, Overview of Docker Compose

This is what you use when you have an application made up of multiple microservices, databases and other dependencies. The docker-compose.yml allows you to configure all those services in one place and start them all with a single command. I’ll cover Docker Compose in much greater detail in the follow up blog post.

In years past, Docker Machine was more popular than it is now.

“Docker Machine is a tool that lets you install Docker Engine on virtual hosts, and manage the hosts with docker-machine commands.” — Docker, Docker Machine Overview

It’s fallen by the wayside a bit as Docker images have become more stable on their native platforms, but earlier in Docker’s history it was very helpful. That’s about all you need to know about Docker Machine for now.

The final tool, Docker Swarm, won't be covered in detail. But for now, Docker Swarm

“creates a swarm of Docker Engines where you can deploy application services. You don’t need additional orchestration software to create or manage a swarm” — Docker, Swarm Mode Overview

These are the tools you’ll become familiar with, and now, we can talk about the Dockerfile.

The Dockerfile — where it all begins

Docker is a powerful tool, but its power is harnessed through the use of things called Dockerfiles.

A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image. Using docker build users can create an automated build that executes several command-line instructions in succession. — Docker, Dockerfile Reference

A Docker image consists of read-only layers each of which represents a Dockerfile instruction. The layers are stacked and each one is a delta of the changes from the previous layer.

This is what I was talking about above, when a Docker container starts up, it needs to be told what to do, it has nothing installed, it knows how to do nothing. Truly nothing.

The first thing the Dockerfile needs is a base image. A base image tells the container what to install as its OS — Ubuntu, RHEL, SuSE, Node, Java, etc.

Next, you’ll provide setup instructions. These are all the things the Docker container needs to know about: environment variables, dependencies to install, where files live, etc.

And finally, you have to tell the container what to do. Typically it will be running specific installations and commands to the application specified in the setup instructions. I’ll give a quick overview of the most common Dockerfile commands next and then show some examples to help it make sense.

Dockerfile Commands

Below, are the commands that will be used 90% of the time when you’re writing Dockerfiles, and what they mean.

FROM — this initializes a new build stage and sets the Base Image for subsequent instructions. As such, a valid Dockerfile must start with a FROM instruction.
RUN — will execute any commands in a new layer on top of the current image and commit the results. The resulting committed image will be used for the next step in the Dockerfile.
ENV — sets the environment variable <key> to the value <value>. This value will be in the environment for all subsequent instructions in the build stage and can be replaced inline in many as well.
EXPOSE — informs Docker that the container listens on the specified network ports at runtime. You can specify whether the port listens on TCP or UDP, and the default is TCP if the protocol is not specified. This makes it possible for the host and the outside world to access the isolated Docker Container.
VOLUME — creates a mount point with the specified name and marks it as holding externally mounted volumes from the native host or other containers.

Ok, now Dockerfile commands have been defined, so let’s look at some sample Dockerfiles. If you want to learn more about Docker commands, I’d highly recommend the Dockerfile documentation — it’s very well written.

Dockerfile Examples

Here are a few sample Dockerfiles, complete with comments explaining each line and what’s happening layer by layer.

Node.js Dockerfile Example

# creates a layer from the node:carbon Docker image
FROM node:carbon

# create the app directory for inside the Docker image
WORKDIR /usr/src/app

# copy and install app dependencies from the package.json (and the package-lock.json) into the root of the directory created above
COPY package*.json ./
RUN npm install

# bundle app source inside Docker image
COPY . .

# expose port 8080 to have it mapped by Docker daemon
EXPOSE 8080

# define the command to run the app (it's the npm start script from the package.json file)
CMD [ "npm", "start" ]

Java Dockerfile Example

# creates a layer from the openjdk:8-jdk-alpine Docker image
FROM openjdk:8-jdk-alpine

# create the directory for where Tomcat creates its working directories
VOLUME /tmp

# copy the project JAR file to the container renamed as 'app.jar'
COPY build/libs /app

# execute that JAR in the entry point below
ENTRYPOINT ["java", "-Djava.security.egd=file:/dev/./urandom", "-jar", "/app/java-example.jar"]

Python Dockerfile Example

# creates a layer from the ubuntu:16.04 Docker image 
FROM ubuntu:16.04

# adds files from the Docker client’s current directory
COPY . /app

# builds the application with make 
RUN make /app

# specifies what command to run within the container
CMD python /app/app.py

Jenkins Dockerfile Example

# creates a layer from the jenkins:lts Docker image
FROM jenkins/jenkins:lts

# sets user to root (because Docker always runs as root and Jenkins needs to know this)
  USER root
 
# add and install all the necessary dependencies
 RUN apt-get update && \
 apt-get install -qy \
   apt-utils \
   libyaml-dev \
   build-essential \
   python-dev \
   libxml2-dev \
   libxslt-dev \
   libffi-dev \
   libssl-dev \
   default-libmysqlclient-dev \
   python-mysqldb \
   python-pip \
   libjpeg-dev \
   zlib1g-dev \
   libblas-dev\
   liblapack-dev \
   libatlas-base-dev \
   apt-transport-https \
   ca-certificates \
   zip \
   unzip \
   gfortran && \
 rm -rf /var/lib/apt/lists/*
 
# install docker
 RUN curl -fsSL get.docker.com -o get-docker.sh && sh get-docker.sh
 
# install docker compose
 RUN curl -L https://github.com/docker/compose/releases/download/1.8.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose && \
     chmod +x /usr/local/bin/docker-compose

# install pip for python
 RUN pip install cffi --upgrade
 RUN pip install pip2pi ansible==2.0
 
# copy groovy executors and plugins for jenkins and run the plugins
 COPY executors.groovy /usr/share/jenkins/ref/init.groovy.d/executors.groovy
 COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
 RUN /usr/local/bin/plugins.sh /usr/share/jenkins/ref/plugins.txt
 
# add the jenkins user to the docker group so that sudo is not required to run docker commands
 RUN groupmod -g 1026 docker && gpasswd -a jenkins docker
 USER jenkins

The commands aren’t really that hard or complicated, once you see the files broken down like this.

Images vs. Containers

The terms Docker image and Docker container are sometimes used interchangeably, but they shouldn’t be, they mean two different things.

Docker images are executable packages that include everything needed to run an application — the code, a runtime, libraries, environment variables, and configuration files.

Docker containers are a runtime instance of an image — what the image becomes in memory when executed (that is, an image with state, or a user process).

<figcaption>Examples of Docker containers. Each one comes from a specific Docker image.</figcaption>

</figure>

In short, Docker images hold the snapshot of the Dockerfile, and the Docker container is a running implementation of a Docker image based on the instructions contained within that image.

We clear? Cool.

Docker Engine Commands

Once the Dockerfile has been written the Docker image can be built and the Docker container can be run. All of this is taken care of by the Docker Engine that I covered briefly earlier.

A user can interact with the Docker Engine through the Docker CLI, which talks to the Docker REST API, which talks to the long-running Docker daemon process (the heart of the Docker Engine). Here’s an illustration below.

<figcaption>The CLI uses the Docker REST API to control or interact with the Docker daemon through scripting or direct CLI commands. Many other Docker applications use the underlying API and CLI as well.</figcaption>

</figure>

The CLI uses the Docker REST API to control or interact with the Docker daemon through scripting or direct CLI commands. Many other Docker applications use the underlying API and CLI as well.

docker build — builds an image from a Dockerfile
docker images — displays all Docker images on that machine
docker run — starts container and runs any commands in that container
there’s multiple options that go along with docker run including
-p — allows you to specify ports in host and Docker container
-it—opens up an interactive terminal after the container starts running
-v — bind mount a volume to the container
-e — set environmental variables
-d — starts the container in daemon mode (it runs in a background process)
docker rmi — removes one or more images
docker rm — removes one or more containers
docker kill — kills one or more running containers
docker ps — displays a list of running containers
docker tag — tags the image with an alias that can be referenced later (good for versioning)
docker login — login to Docker registry

These commands can be combined in ways too numerous to count, but here’s a couple simple examples of Docker commands.

docker build -t user1/node-example .

This says to Docker: build (build) the image from the Dockerfile at the root level (. ) and tag it -t as user1/node-example. Don’t forget the period — this is how Docker knows where to look for the Dockerfile.

docker run -p 3003:8080 -d user1/node-example

This tells Docker to run (run) the image that was built and tagged as user1/node-example, expose port 3003 on the host machine and look for port 8080 inside the Docker container (-p 3003:8080), and start the process as a background daemon process (-d).

That’s how simple it can be to run commands using the Docker CLI. Once again, the Docker documentation is well done.

Conclusion & Part Two: Docker Compose

Now you’ve seen the tip of the iceberg of Docker. It’s a lightweight, isolated runtime environment known as a ‘container’ that you can spin up and configure just about anyway you like with the help of a Dockerfile.

Stay tuned for part two of this series where I’ll dive into the Docker Compose tool that lets you configure and run multiple applications with just one file and a single start command. It’s pretty darn cool.

Thanks for reading, I hope this gives you a better understanding of the basics of Docker and its power.

References & Further Resources

React, SVG Images and the webpack Loader to Make Them Play Nice

Fri, 08 Jun 2018 00:00:00 GMT

<figcaption>Photo by <a href="https://unsplash.com/photos/iacpoKgpBAM?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Goran Ivos</a> on Unsplash</figcaption>

</figure>

Introduction

This isn’t a new problem - the dilemma of getting SVGs to correctly render with React. But it is a problem that I hadn’t had to deal with and overcome until recently, and I’d like to share my experiences in the hope that it saves someone else a few hours of searching Github, StackOverflow and npm.

At my current company, I’ve worked for two separate development teams, both of whom use JavaScript microservices for their UIs: one uses AngularJS, the other JavaScript MVC. All the backend services are Java Spring Boot applications.

Recently, the news broke that Google was going to end long term support for AngularJS v1.5 - the version we were running on.

It quickly became clear we had three options:

Upgrade the system to AngularJS 1.7 (which would continue to have long term support for 3 more years),
Upgrade our application to Angular 2+,
Or migrate to a competitor JavaScript framework: ReactJS.

One of my colleagues and I were tasked with figuring out which was the best direction to take. This exploration was a process which deserves its own blog post, but suffice it to say, I was most in favor of venturing into React (I liked the syntax better for one). After we’d looked at our options and tinkered, we decided to see if we could make a small proof-of-concept (POC) in React work with our existing app (because we'd need the two UIs to route back and forth to each other until the migration was complete).

About 4 hours after starting our React POC, we had a working, albeit not very pretty, app. We added Jest and Enzyme unit tests, Cypress end-to-end tests and more. But for almost two weeks one thing eluded us: getting an SVG graphic to render correctly in the DOM.

Today, I'll show you how I got SVGs working with a React application.

Why SVGs? What’s the Benefit? Why Not Just Make It Into a PNG?

Great question. The benefits of using SVGs are many, and in case you don’t know, SVG stands for Scalable Vector Graphics. While the textbook definition sounds dense:

“SVG is an XML-based vector image format for two-dimensional graphics with support for interactivity and animation” — Wikipedia, Scalable Vector Graphics

They basically take the place of traditional images in other formats like JPEG, PNG, TIFF or GIF.

Here are a few reasons why SVGs are superior to the other image formats:

Crystal clear resolution no matter the screen size, zoom level or image size — since SVGs are built on vector paths, shapes and fills, they can scale to any size with no loss of clarity. PNGs, on the other hand, are based on pixels, which won’t look sharp and clear at all times (especially on retina display devices).
Load speed is much faster — if you’re using high res images on your site to cater to those retina displays, the PNG files can be pretty sizable, and larger files render more slowly. SVGs are just code, which means smaller file sizes (unless you’re loading something very detailed with a million vector paths).
Animation is possible — unlike PNG files which are static: what you see is what you get - SVGs can be animated and styled with CSS. Animations, different colors, changes in state (like mouse hover or click) - all these things can be done to SVGs.
Accessibility and SEO — Google indexes SVGs, which means better SEO scores for your site and better accessibility for people with vision impairments visiting your site.

Right. So SVGs It Is, Now How to Use Them In React

For my project, I was attempting to recreate one of the simpler screens of the existing AngularJS application in React. This screen contained four big, clickable cards that would take users to other various screens, and each card had a nice SVG file associated with words.

So I copied the SVG files from the original project into the React app, imported the files into the component that would render them, and told the DOM to render them:

<Image source={require("../img/key.svg")} />

In the JSX, I added the CSS styling, waited for webpack to do its magical parsing and... nothing. Well, not quite nothing, errors in the console about unrecognized file types, but certainly no SVG key-shaped icon displayed in the UI.

And thus began the search for the library I needed to render SVGs correctly in React. This was a days long process. If you’re unaware there’s approximately 7,845,923 npm modules that all claim to make React play nicely with SVGs. Some of them probably work, the first 10 or so I tried, did not.

Then… SVG Inline Loader

Then I found it. An npm module aptly named svg-inline-loader, it had decent documentation, it had 3,200 weekly downloads, it had 325 stars on Github, it was developed and maintained by the core team that built webpack - it seemed promising.

So I gave it a shot and saved it to my devDependencies in the package.json file.

$ npm install svg-inline-loader react-inlinesvg --save-dev

Then added the configuration object to the webpack.config.js file:

{
    test: /\.svg$/,
    loader: "svg-inline-loader"
}

And I was good to go. One simple update to webpack, a small adjustment to the JSX syntax and...

import React, { Component } from 'react';
import KeyImage from '../img/key.svg'; 
import SVG from 'react-inlinesvg';
class Card extends Component(){
return(
    <div>
      < /* other UI stuff here */ > 
      <SVG src={KeyImage} />
    </div>  
  )
}

BOOM — SVG displayed!

I’d also like to add that this npm module has some pretty nifty extra features: it can remove extra tag info from SVGs (which happens a lot when they’re generated with Sketch or Adobe), and it can add classes or IDs to prevent SVG collisions. Nice.

And that was it. No CSS changes, no super weird syntaxes, nothing like that. Just svg-inline-loader, an extra module loader for webpack, and the problem was solved.

Thanks for reading, I hope it helps you like it helped me.

References & Further Resources

Scalable Vector Graphics definition
svg-inline-loader
5 reasons you should be using SVGs over PNGs

Leveraging a Spring Cloud Config Server in a Node.js App's Feature Toggles (Pt 2)

Sat, 02 Jun 2018 00:00:00 GMT

</figure>

Introduction

As I showed in my first post about the Spring Cloud Config server, setting up a configuration server for a Java Spring Boot application and then leveraging that server to provide environment variables to existing Spring Boot projects is really handy.

The next trick was figuring out if it was possible to use that very same config server in a Node,js project.

To give you a quick rundown, my team worked on a microservice-based application architecture, meaning, we had up to thirteen separate Java services and one Node service all running to power our application.

It made total sense to go with a Java-based config server, since: thirteen Spring Boot microservices, but when our business partners decided they wanted to implement feature toggles, it became a priority to get the config server working in the Node project as well.

What is a Feature Toggle, and Why Do I Need It?

Let me set the stage by giving you a quick run down of what a feature toggle is and why it’s so beneficial for developers, business partners and customers.

A feature toggle is a technique in software development that attempts to provide an alternative to maintaining multiple source-code branches (known as feature branches), such that a feature can be tested even before it is completed and ready for release. A feature toggle is used to hide, enable or disable the feature during run time. For example, during the development process, a developer can enable the feature for testing and disable it for other users.

— Wikipedia, Feature Toggle

Basically, a feature toggle allows developers to introduce new features to users in a more Agile development fashion.

With feature toggles, devs can continue to maintain just one codebase but show different views of the UI to users in production versus the development team in lower life cycles, based on if those toggles are set to ‘on’ or ‘off’.

They also allow our business and UX partners to validate early on, if new features are beneficial to our users, or if they need to reevaluate how we’re approaching a solution.

Let me make one thing crystal clear though: feature toggles are not meant to be permanent solutions. They are temporary, meant to remain only until the feature has been completed and validated that it’s useful or meets whatever other criteria was set for success or failure. Then the code around the feature toggle should be removed to keep the codebase clean and as free of technical debt as is possible.

For my specific situation, our feature toggle was needed to hide a button that didn’t have full functionality yet because the backend service to support it wasn’t finished, and hide the radio buttons from users. They wouldn’t be needed until the button and service were done.

<figcaption>Feature toggle view for users: no button and no checkboxes.</figcaption>

</figure>

Above is the UI view users in production would see: no buttons, no checkboxes. Below is the UI view developers working on the feature need to see: buttons and checkboxes present.

<figcaption>Feature toggle view for devs: buttons and checkboxes.</figcaption>

</figure>

Let's look at how we got these two views simultaneously.

Setting Up Your Config Server Properties

Before we get to the Node config server, let’s first set up the new file with the following naming conventions in our config server properties repo:

[application name]-[life cycle profile].yml
ex. my-ui-app-to-config-QA.yml

Within this YAML file, you’ll be able to add your configuration properties for your UI feature toggle. Here’s all I had to include for my feature toggles.

modifyDatesToggle: true

Commit this to a Github repo, and we’re set there. You can see an example of the config properties repo here in Github.

nodecloud-config-client Node.js Set Up

To leverage the existing Spring Cloud Config server set up with our Node frontend application, I chose the nodecloud-config-client on npm, as it’s a fairly well documented Spring Cloud Config Server client written for JavaScript.

I should mention, this is not to be confused with node-cloud-config-client or cloud-config-client — really, there’s a million versions of config client packages written for Node.js, but the one listed above and in the resources link is the one I used.

So to get down to business, after saving the npm package to our UI’s package.json dependencies, add it to the server.js file (or wherever your Node server’s configuration is located).

<figcaption>See the client variable at the bottom — that’s where the config client comes in to play.</figcaption>

</figure>

If you’re developing in multiple environments like my team did (local, QA, Q1, production, etc.), add a variable to access the Spring Cloud config server for all stages of the development process (these environment variable services are how we connected to it in Pivotal Cloud Foundry). I mentioned these back in my previous post, if you’d like more info.

<figcaption>The configServerUrl goes through a series of checks for environment variables, and if it finds none of those (like during local development), it defaults to the hard coded URL. I used the same environment variables and logic for the configServicesProfile and featureToggle.</figcaption>

</figure>

Next, add a variable to account for where the config services profile will be located (for local development I had it default to QA). And I added a variable to check for enabled/disabled feature toggles.

The below code connects to the config server (when feature toggles are enabled), and maps any feature toggles found to an object.

{ modifyDatesFeature : true }

Then that object is sent to the rest of the UI with the app.get endpoint simply called "/featureToggles".

<figcaption>Feature flag variable, call to the client to access the config server with the correct environment (QA in my case), and a check if the featureToggle is ‘on’, map the properties that are available to it. Below is the endpoint to call the features.</figcaption>

</figure>

Great. The Node server is set up, and you can get the features toggles (or whatever else you’re storing in the config server) just by changing a few variables.

Connecting Your Client Side UI Framework to Your Node Server

The first thing you’ll need to do on the frontend is make an AJAX call to the Node server to check for any existing feature toggles. The framework of the app implementing this was JavaScript MVC (an older framework popular years ago, I'd never heard of it before I worked with it either), and here’s an example of what the call looked like.

</figure>

In the JavaScript file that’s actually concerned with feature flags (for me, it was a dates.js file), I imported the feature toggle AJAX call and created a function to check for the specific feature flag associated with this functionality: modifyDatesToggle.

<figcaption>The feature flag was imported at the top of the file, then I call to the feature flag endpoint to see if there’s anything there the file needs to be aware of. If the feature flag matches the variable I named “modifyDatesToggle”, it gets pulled in and applied to the file.</figcaption>

</figure>

Finally, inside your JavaScript file using the feature toggle, group the code, wherever possible according to if the feature toggle is enabled, like so.

<figcaption>Example 1: This check displays the check boxes if the modifyDatesToggle is “on.”</figcaption>

</figure>

The thinking goes, that by wrapping all the feature enabled code inside of the feature toggle check, even if the call to the config server fails for some reason, users will only ever see the code that should be deployed to production. This gives us extra protection by not being as dependent on more variables than necessary.

But Wait, There’s More — Enabling End-to-End Tests with Feature Toggles

There’s one more thing I wanted to include in this blog: how I used the same feature toggle logic to run or skip some of the Protractor end-to-end tests.

If you’re using a configuration file for Protractor, and I don’t know why anyone wouldn’t, there’s a cool feature that lets you make your own custom parameters. I took a page out of my own book where the feature toggle is set in the server.js file, and did a similar set up in the Protractor config file: if environment variables are passed in for the feature toggle, the tests that concern that feature will run, if it’s not, they’ll be ignored — I’ll show you the test syntax in a moment, as it’s a little unconventional.

<figcaption>Here’s the configuration that all the Protractor tests use. Params takes in the modifyDatesToggle object and if it exists, it sets the params, if it doesn’t, it sets it to null.</figcaption>

</figure>

When you’re running the tests locally, you can actually pass the params in through the command line like this:

$ protractor e2e/conf.js browser.params.modifyDatesToggle=‘true’

To implement the feature toggle in the actual end-to-end tests, inside of each test that requires a feature toggle check, add an if statement right after the ‘it’ declaration, checking if the feature toggle is null or not. As I said, this is a little weird to see, but it works.

<figcaption>An example of a Protractor end-to-end test with a feature toggle check. Not all the tests need this check — just the ones that require the feature toggle to be either off or on.</figcaption>

</figure>

If the feature toggle is not enabled, the test will be skipped, but show up as if it passed in the console (eliminating failing tests because the proper environment wasn’t present for the test to use).

If the feature toggle is enabled, the test will run as normal. This prevents us from having to go through the tests one by one and either x them out to ignore them or remove the x so they’ll run depending on what the desired functionality is. And once it’s time to remove the feature toggle, it’s easy enough to either remove the unnecessary tests from the spec file or just x them out, depending on your team’s preference for keeping tests even after functionality has changed. (Personally, I would remove it.)

Above, I showed how to run the feature toggle tests from the command line, but it can also be run programmatically when the application is being deployed.

Segue: for my team personally, we use Jenkins for all our builds. That’s a whole other set of blog posts, but suffice it to say, while the build is running all the unit test are run, all the end-to-end tests are running, all the rest of the checks we have in place before declaring a feature is tested and ready for our business team to accept happens.

Back to how this matters: since Jenkins is our job runner, we can pass the environment variables for the feature toggle tests to Jenkins through a Jenkinsfile.

<figcaption>All that’s needed is the simple line at the bottom: env.MODIFY_DATES_TOGGLE=true</figcaption>

</figure>

And that should be all you need to run your end-to-end tests — feature toggles on or off.

Automating The Feature Toggles

As I said before, my team hosted our applications on Pivotal Cloud Foundry, and in PCF, we use things called VCAP services to hold our environment variables. Through manifest files we could bind these services to our applications and pass in environment variables — variables like if feature toggles should be turned on or off. See where I’m going here?

The check for the feature toggle environment variable makes deploying the app in an automated fashion easier. By including a simple environment variable (or not) to the manifest that deploys with each different stage of development, it’s easy to tell the application to check for settings in the config server (and employ them if need be) or not.

<figcaption>QA manifest screenshot.</figcaption>

</figure>

QA manifest: note the SPRING_PROFILES_ACTIVE pointing to QA, FEATURE_TOGGLE environment variables with ‘on’.

<figcaption>Production manifest screenshot.</figcaption>

</figure>

Production manifest: no feature toggle in sight, and the SPRING_PROFILES_ACTIVE is set to production.

Conclusion

There you have it. Now, you have seen how you can leverage a Spring Cloud Config server you’ve built with your Node.js application. Not only that, you can also configure end to end tests to run with the feature toggle on or off as well.

If you’ve found other ways to implement feature toggles or other nifty things like this in your own JavaScript projects, I’d love to hear about them.

Thanks for reading!

References & Further Resources

Why a Spring Cloud Config Server is Crucial to a Good CI/CD Pipeline (Pt 1)

Sat, 26 May 2018 00:00:00 GMT

</figure>

Introduction

Before I began developing large-scale, enterprise level software, I didn’t fully comprehend the value of things like integration tests, automated build processes and shared libraries.

For the small applications I built during my coding bootcamp, it was easy for me to do things like hard code URLs so my frontend application could talk to its backend counterparts and their databases.

I only had two environments: my local development environment on my laptop and my AWS production environment where I hosted my projects portfolio, and it was a fairly simple, albeit entirely manual, process (with a good bit of trial and error) to get the pieces of the puzzle connected.

Fast forward several months to my software engineering job of maintaining and improving upon an application backed by no less than 13 microservices. No, I’m not joking. Yes, all 13 are critical to this application working.

Did I mention that in my team’s test driven development (TDD), agile process we also have 4 different development environments before a new feature makes it to production? Yep: a local development space, a QA space, a Q1 space, an acceptance space and finally our production space.

So, think about it: does it make sense to have to manually update each and every one of these services as they move through the development process on their way to production? Of course it doesn’t.

You need a centralized, easy-to-automate way to update those environment variables. This is where a cloud configuration server can shine.

To give you a quick overview, cloud config servers are designed to:

“Provide server and client-side support for externalized configuration in a distributed system. With the Config Server you have a central place to manage external properties for applications across all environments.”

- Spring Cloud Config

Basically, a config server allows you to externally store variables your application will need to run in all environments, regardless of lifecycle, and update them in one, centralized place.

<figcaption>Behold, the Spring Cloud.</figcaption>

</figure>

The first config server I was introduced to is the Spring Cloud Config server, since my team is responsible for a full stack application composed of many Java Spring Boot backend microservices and a JavaScript Node.js frontend microservice.

Setting Up Your Config Server

Setting up the config server is actually pretty easy.

To get started, you can go the Spring Initializr site, add Config Server as a dependency to the project and click the "Generate Project" button in the browser.

I’ll also add that, at least at the time I’m writing this, the Spring Cloud Config Server is not compatible with Spring Boot version 2 (I found this out the hard way), so choose the highest Spring Boot version 1 snapshot you can.

<figcaption>This is all you need to get your Spring Cloud Config Server running: the Config Server dependency. Note the version of Spring Boot is 1.5.14, not 2.0.2, which is the latest version it automatically defaults to.</figcaption>

</figure>

Once you’ve downloaded the project and opened it up in your IDE of choice (mine is IntelliJ), you can go straight to the main application file, add the @EnableConfigServer and @SpringBootApplication Spring Boot annotations, and you’re almost ready to go.

Below is my actual main method file, truly, that’s all it needs.

package com.myExampleConfigServer; 
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.config.server.EnableConfigServer; 

@EnableConfigServer
@SpringBootApplication

public class MyExampleConfigServerApplication {  
  public static void main(String[] args) {
    SpringApplication.run(MyExampleConfigServerApplication.class,  args); 
    }
}

The other file you’ll need to configure is your application.yml file in your resources/ folder. This file will be where you set up your cloud config server’s access to GitHub (and the GitHub files it will use to provide application configuration variables).

As you can see, there’s a spot for your GitHub URL (which I’ll come back to in a minute), username and password.

Since our applications are hosted on Pivotal Cloud Foundry, we use what are called VCAP services (it stands for VMWare Cloud Application Platform). It’s a fancy way of saying a service where environment variables are stored that only people (and applications) who have access to that space within PCF can access. This lets us put sensitive information like service accounts and passwords somewhere besides a public GitHub repo where the credentials are available for the config server, but not available for anyone without authorization.

Here’s my application.yml. With these two files configured and a cloud platform to host your server, you’ll be ready to start using your config server.

application.yml

server:  
  port:
    8888

spring:  
  application:    
    name: myexample-config-server  

cloud:    
  config:      
    server:        
      git:          
        uri: ${vcap.services.config-service.credentials.url}
        username: ${vcap.services.config-service.credentials.user}          
        password: ${vcap.services.config-service.credentials.token}

If you’d like to see an example of the config server app, I’ve put a starter project here in GitHub.

Ok, so you’ve got your config server. Great, now it’s time to set up your config server properties — the files the config server will actually access to pull environment variables for your projects.

Setting Up Your Config Server Properties

This part is a cinch. Create a totally empty new repo in GitHub, and then create a new file with the following naming conventions:

[application name]-[life cycle profile].yml

ex. my-app-to-config-QA.yml

Within this YAML file, you’ll be able to add your configuration properties for your application. Here’s some sample info you might include in the YAML.

my-app-to-config-QA.yml

configurations:
  featureToggleFlag: true
  my-custom-flag: false
  sampleCronJob: "0 0 1 * * *"
  sampleUrl: http://google.com

Commit this file to a GitHub repo, copy that repo URL and place it in the Cloud Config Server URI in your config server’s application.yml. Now the server knows where to look in GitHub for the config files you want to use.

You can see an example of a config properties repo here in GitHub.

Connecting Your Java Spring Boot Application to Your Config Server

With both your config server and your properties you want the server to provide to your project, you need your project to be able to get those properties when your Spring Boot service starts up.

Again, there’s not a lot of overhead to getting this working. In the build.gradle file of your app that needs these config server values, you’ll need the following dependencies:

build.gradle

compile group: 'org.springframework.cloud', name: 'spring-cloud-starter-config', version:'1.3.3.RELEASE'
compile group: 'org.springframework.boot', name: 'spring-boot-configuration-processor', version: "${springBootVersion}"

Then, you’ll create a bootstrap.yml file which will live alongside your application.yml file in your src/main/resources/ folder, and it will contain information pointing to the config server’s location wherever it’s being hosted. It will look something like this:

bootstrap.yml

spring:
  application:
    name: ##APPLICATION NAME GOES HERE##
  cloud:
    config:
      uri: https://myexample-config-server.non-prod.com
---
spring:
  profiles: production
  cloud:
    config:
      uri:  https://myexample-config-server.prod.com

Next, you’ll go to whichever files actually need the config properties and add the annotations @Configuration and @ConfigurationProperties(prefix=”configurations") (or whatever you’ve named your config properties in the config properties YAML).

Here's an example:

XyzProperties.java

@Configuration
@ConfigurationProperties(prefix = "configurations")
public class ConfigProperties {
    
}

If you notice, the properties are prefixed by configurations. To bind these properties to the ConfigProperties class, you'll first need to add a "prefix" to the @ConfigurationProperties annotation like so:

To bind the “featureToggleFlag” property, add the following member variable. along with the getter and setter. Or use Lombok’s @NoArgsConstructor and skip the getters and setters - your choice.

XyzProperties.java

@Configuration
@ConfigurationProperties(prefix = "configurations")
public class ConfigProperties {
   private boolean featureToggleFlag;
   
   //Add Getters and Setters here if desired...
    
}

If you match the member variable name with the actual property name in the configuration properties, Spring will automatically bind it to the member variable. So far so good.

The very last thing you must do before starting up your service is ensure that you set SPRING_PROFILES_ACTIVE to the correct config properties environment. The current values would be “QA” for the QA environment, “Q1” for the Q1 environment, “prod” for production and so on. Be warned, these values are case sensitive, so however you’ve named them in your config properties file, it must be exactly the same in the "Active Profiles" input in IntelliJ's start script setup.

In IntelliJ, this can be done by clicking the "Edit Configurations" property when setting up a Spring Boot project to run, and adding the correct value to the "Active Profiles" input midway down the config modal.

<figcaption>Inside your IntelliJ run configuration, add this line in the "Active Profiles" input.</figcaption>

</figure>

And that’s it.

Now, when you start up the Spring Boot service, you should see Spring Profile being set in the logs, and to be sure, you might also put in some System.out.println messaging to let you know if it’s successfully reached the config server and acquired the properties needed.

Fantastic, we've built a config server, we've moved environment variables out of your Spring Boot project to a centralized location, so changes can be made quickly and we've configured a Java service to pull those variables from the config server. Nicely done.

Conclusion and Part Two: The Config Server and Node.js

But, how do you do the same thing with a JavaScript / Node.js project? Can you use the same Spring Boot config server? Or do you have to set up a separate Node-based config server?

You can use the existing one, and I’ll show you how to set it up, and how you might use it in a JavaScript project to enable and disable feature toggles for faster feature deployments in my next post.

Thanks for reading!

References & Further Resources

What is Graphql, Really?

Sat, 12 May 2018 00:00:00 GMT

</figure>

I’ve been hearing about GraphQL for months now, but for the longest time I really didn’t understand what it was.

Is it a new query language? A new type of database? Some other JavaScript black magic? Nope, none of the above.

GraphQL is a new API standard that provides a more efficient, powerful and flexible alternative to REST.

— How to GraphQL

It’s an alternative to REST, so what? Why should I use GraphQL instead?

Great question.

Here’s what sets GraphQL apart from traditional REST endpoints: with GraphQL, a client can specify exactly what data it wants (from multiple sources), and get back just that from a single endpoint.

Traditional REST endpoints return fixed data structures, usually with extra info clients don’t need, and they require multiple endpoints (and HTTP calls) to get back all the necessary information. GraphQL does not.

Here's an example of two traditional HTTP calls to find students and their classes

// Call 1: HTTP get to: 'students/<id>' 
// Returns:
{ "student": 
  { 
    "id": 1, 
    "firstName":"Jane", 
    "lastName":"Smith"
  } 
}
// Call 2: HTTP get to: 'students/<id>/classes'
// Returns:
{ "classes":
  [{
    "id": 1, 
    "className":"math"
  }, 
  {
    "id": 1, 
    "className":"english"
  }]
}

/* Here is the same information as one GraphQL query */*
// Query:
query { Student( id:1 ){ 
    firstName 
    lastName 
    classes { 
      className
    }
  }
} 
// Returns:
{ "data": 
  { "Student": 
    { "firstName":"Jane", 
      "lastName":"Smith", 
      "classes": 
      [{
        "className": "math"
      }, 
      {
        "className":"english"
      }]
    }
  }
}

One thing that cannot be stressed enough is that GraphQL is not a query language for databases. GraphQL is a query language for APIs.

This makes it database agnostic, and makes it possible to be used in any context where an API is used.

And no, contrary to the fact that whenever GraphQL is mentioned, it seems to be in tandem with React (since Facebook invented both), GraphQL can be used everywhere a client communicates with an API: there’s server libraries for C, Go, Erlang, Java and more.

JavaScript is by far the most popular language to utilize with GraphQL, and it has some really excellent server packages that work in conjunction to make GraphQL easily implementable with projects.

It can be run straight from the command line with Node.js when Graphql.js is installed, but it really shines when it’s integrated into a project with the GraphQL API server sitting as a layer between the client frontend and the server backend and database(s).

The implementation I’ve used is Apollo, and it integrates easily with various Node.js HTTP frameworks like Express, KOA, Hapi, etc. But that’s for another post.

Ok, you’ve piqued my interest. So how does GraphQL work?

GraphQL works based on strongly typed schemas.

In short, everything that you want GraphQL to return to you, must be explicitly defined in the schema (which some developers like and others feel like it’s making them define data twice).

Schemas: How GraphQL Defines and Structures the Data it Receives

// **** Simple Schema Example for a Type Called Student ****
type Student { 
  firstName: String! // (the exclamation mark means it's required)
  lastName: String!
}

The Schema Definition Language (SDL) is the syntax for writing schemas, and queries and mutations are composed to get back the required information from the server. Since the structure of the data returned from the single GraphQL endpoint is completely flexible, the query has to be extremely specific in what it requires.

If something is specified in the schema but not returned, the response will contain a null value for that field, but it will not throw an exception as sometimes happens with REST endpoints. However, if a specific field is not specified but is needed, the client won’t receive it until the schema is updated to include that field.

Queries: How GraphQL Asks for Data

Here's a basic query asking the server for all students

// **** Basic Query Asking the Server for All Students ****
{ allStudents 
  {
    firstName 
    lastName
  }
}
// Returns: 
{"allStudents":
  [{
    "firstName":"Jane", 
    "lastName":"Smith"
  },
  {
    "firstName":"John", 
    "lastName":"Doe"
  }]
}

Mutations: How GraphQL Manipulates Data

In addition to queries, GraphQL uses something called mutations to manipulate data stored in the backend. There’s generally three kinds of mutations to:

create new data
update existing data
delete existing data

This mutation create a new student in the database

// **** A Create Mutation to Make a New Student ****
mutation { createStudent(firstName: "Robert", lastName: "Johnson") 
  { 
    firstName 
    lastName
  }
}

// The server response would look like this:

"createStudent": { 
  "firstName":"Robert", 
  "lastName":"Johnson"
}

Between queries to read data and mutations to manipulate data, all the CRUD functionality supplied by traditional REST endpoints is handled by GraphQL.

Regardless of the way you’re implementing the GraphQL server (connected to a single database, over a number of third party legacy systems or as a hybrid of the two), when a query arrives at the GraphQL server, it resolves the query by reading the payload and fetching the data. Then it uses the schema to return the data in the correct format to the client.

This clearly defined schema structure is what helps GraphQL to be transport-layer agnostic (TCP, Websockets, etc. — it’s all good) and database agnostic (MySQL, MongoDB, Oracle, etc.).

Likewise, if GraphQL is being implemented over multiple systems (as is likely the case in larger corporations with lots of sprawling legacy systems and APIs), it can neatly pull only the requested information from all the various endpoints, and resolve it into one, clean response, laid out by the schema.

Resolvers: How GraphQL Fetches the Data for Its Query (or Mutation) Field

Each of the fields that makes up a piece of the GraphQL query function is called a resolver, and the sole purpose of the resolver is to fetch the data for its field.

// **** Sample Query and the Resolvers Corresponding to Each Field ****
query { Student(id:1){ 
    firstName 
    lastName 
  }
}
// Resolvers:
Student(id: String!): Student
firstName: (student: Student!) String
lastName: (student: Student!) String

GraphQL Client Libraries

GraphQL is a big step forward for frontend developers because instead of having to imperatively fetch data through:

building and sending an HTTP request (fetch in JavaScript),
receiving and parsing the response,
storing the data (locally or persisted),
and finally displaying the data in the UI,

Clients can declaratively fetch the data by:

describing the data requirements,
displaying the returned data in the UI

Complexity and data storage are pushed to the server-side of the application, which is better suited to take care of the heavy computation work, and the frontend is free to do what it was designed to do: show that data to users quickly in a way they can understand.

What I’ve described above is what GraphQL libraries like Apollo and Relay were designed to do. They abstract away the infrastructure implementation of connecting the backend services so developers can focus on the important parts of their application.

More About GraphQL

If you wanted to, you could stop right here with a much better understanding of what GraphQL is, but if you’d like to know a little more about the benefits on both the client and server side with GraphQL, read on.

GraphQL and the Client Side

As discussed in the previous sections using the GraphQL API makes frontend development easier with abstractions and helping implement common functionality on the client side.

Here I’ll go into more detail on how it handles “infrastructure” features developers normally want in their apps.

Directly Sending Queries and Mutations Sans HTTP Requests

GraphQL allows for fetching data in a declarative manner. Instead of making low-level HTTP calls, a client can simply query for the data it needs and GraphQL takes care of the request and response handling for you.

View Layer Integrations & UI Updates

Once the GraphQL client has received and handled the response, there’s a number of ways the UI can be updated to reflect the data. If you’re using React, the GraphQL clients make use of higher-order component concepts (when a function takes one component and returns a new component) to fetch the data make it available to your components in their props.

Client Side Caching

Typically, you’ll want to maintain some kind of a cache or data previously fetched from the server, it’s critical to having a good user experience and shorter load times as users interact with the UI. However, storing the data exactly as it’s resolved from the GraphQL client won’t be the most efficient way to hold it for most applications.

Normalizing the data (flattening nested query results) so that the store only contains individual records that can be referenced with a unique global ID is a better option for quickly retrieving only the necessary data when the same query is made in the future.

Validating & Optimizing Queries Based on the Schema

Also, since the schema contains all the information about what a client could do with a GraphQL API, the build environment can parse the GraphQL code that’s in the project and compare it against the schema’s information. This makes for catching errors and typos much earlier in the process easier (and less likely to happen in the hands of users).

Likewise, UI code and data requirements can actually be written side-by-side in some languages like JavaScript, which makes it easier for developers to see that the right data ends up with the right parts of the UI.

GraphQL and the Server Side

Although GraphQL is talked about a lot around the frontend API, the API itself is still implemented on the server side. And it allows the backend developer to focus on describing the data, rather than implementing and optimizing numerous REST endpoints.

GraphQL Query Execution

For starters, GraphQL uses a simple algorithm for how it changes its queries into results. It traverses the query field by field, executing resolvers for each field.

If a parent resolver function is required by a child, the parent query will resolve first and pass its result to the child for use in its own query. Finally, once the execution algorithm is done, it forms all the data into the correct shape and returns it.

// **** Sample Query and Execution Algorithm ****
type Query {
  director(id: ID!): Director
}
type Director {
  movies: [Movie]
}
type Movie { 
  title: String 
  description: String
}

// **** Sample Query to the Server ****
query { director(id: "2wsx3edc") 
  { movies 
    { 
      title 
      description
    }
  }
}

// **** Execution Algorithm Visualized ****
Query.director(root, { id:"2wsx3edc" }, context) -> director
Director.movies(director, null, context) -> movies
for each movie in movies
  Movie.title(movie, null, context) -> title
  Movie.description(movie, null, context) -> description

Batched Query Resolving

GraphQL also offers the ability to make data fetching smarter, if multiple similar calls will be made to the server in one query.

In JavaScript, a utility called DataLoader can be used to wrap the fetching function, which will wait for all the resolvers to run, then make sure to only fetch each item once.

// **** Sample Query and API Calls ****
query{ movies
  { title 
    director
    {
      firstName 
      lastName
    }
  }
}

/* Regardless of the fact that the API is being called for the same piece of data multiple times, the query executes for the director info */
fetch('/directors/1')
fetch('/directors/2')
fetch('/directors/2')
fetch('/directors/1')
fetch('/directors/1')

// **** Sample API Calls with DataLoader****
directorLoader = new DirectorLoader()
// Queue up all the fetches
directorLoader.load(1);
directorLoader.load(2);
directorLoader.load(2);
directorLoader.load(1);

/* Then the loader only makes the necessary amount of calls for each unique piece of information */
fetch('/directors/1')
fetch('/directors/2')

GraphQL Tooling and Its Ecosystem

As I mentioned briefly, GraphQL’s Type System allows us to define the surface area of our APIs and validate queries against a schema. What’s cool, is that GraphQL also allows clients to ask a server for information about its schema, in a move called introspection.

GraphQL Introspection

The designers of a schema obviously know what fields are available, but clients can query GraphQL using the __schema meta field available on the root type of a Query to find out too. This is a powerful technique that can be used to provide many interesting, useful features.

// **** Introspection Query and Return Information ****
query{ 
  __schema { 
    types 
    { 
      name 
    }
  }
}
Schema Definition:
type Query {
  director(id: ID!): Director
}
type Director {
  movies: [Movie]
}
type Movie { 
  title: String 
  description: String
}

// Results:
{
  "data": {
    "__schema": {
      "types": [
        {
          "name": "Query"
        },
        {
          "name": "Director"
        },
        {
          "name": "Movie"
        },
        {
          "name": "ID"
        },
        {
          "name": "String"
        },
        {
          "name": "__Schema"
        },
        {
          "name": "__Type"
        },
        {
          "name": "__TypeKind"
        },
        {
          "name": "__Field"
        },
        {
          "name": "__InputValue"
        },
        {
          "name": "__EnumValue"
        },
        {
          "name": "__Directive"
        },
        {
          "name": "__DirectiveLocation"
        }
      ]
    }
  }
}

GraphiQL & GraphQL Playground

Finally, there are two very useful GraphQL IDEs I’m familiar with. The first is GraphiQL, which is a JavaScript, in-browser IDE that’s easy to install and use in projects (it also comes pre-bundled with Apollo for even easier use).

The second is GraphQL Playground, which can be downloaded as a desktop app or used on its website. It incorporates some of the same components as GraphiQL, but boasts additional features like automatic schema reloading, query history, the ability to work with multiple GraphQL APIs at once, interactive schema documentation and more.

In short, both are excellent tools, that allow you to debug and try queries on a GraphQL server without having to write queries over curl.

Conclusion

I hope this helps shed some light on the hot topic that is GraphQL. This is just the tip of the iceberg in terms of what it can do and how it does it, but there are already tons of great resources out there to learn more and start building your own projects using it.

Personally, I haven’t used it for very long yet, but I’m already enjoying it.

Thanks for reading!

References & Further Resources

JavaScript Array Methods to Make You a Better Developer

Mon, 30 Apr 2018 00:00:00 GMT

</figure>

Arrays are Such a Useful Data Structure in JavaScript

Back when I began my new career as a web developer, the fact that I could loop through an array of objects in JavaScript was very cool.

const greetings = ['Hello', 'Guten tag', 'Salaam', 'Hola', 'Ciao'];

sayHello = (greeting) => {
    for(let i = 0; i < greeting.length; i++){
        console.log(greeting[i]);
    }
};

sayHello(greetings);
// Hello
// Guten tag
// Salaam
// Hola
// Ciao

The classic for loop will always have its place, but there’s a world of other array methods just waiting to be utilized in JavaScript. The right tool for the right job is what I’m constantly trying to keep in mind when solving problems.

Now, having worked at it for almost 2 years, I’ve learned a heck of a lot more, including some much more sophisticated methods for manipulating arrays, which I’d like to share today.

One more thing I’d like to add before diving in, is that, in general, it tends to be a better practice not to alter existing arrays or objects. If you mutate the original item, you run the risk of unforeseen side effects: wherever else in the code that array or object is referenced, the mutated result could be used instead. This can make finding the root cause of a bug caused by the mutated code more difficult than it needs to be, to say the least.

Therefore, the methods I cover are known as pure functions, because instead of changing the original array, they instead make a copy and make the changes to that copy and return it to the scope outside of the method as a new array. Just something to keep in mind as you read through this article.

Map

The first method that will come in extremely handy is Array.map(). The map method takes in an array, and creates a new array with the results of calling a function for every element in that array. It sounds a little complicated but it’s not too bad once you see some examples.

Here’s Array.map() being used on an array of strings:

const greetings = ['Hello', 'Guten tag', 'Salaam', 'Hola', 'Ciao'];

const sayHelloWorld = (hellos) => hellos.map(hello => {
    return `${hello} world`;
});

console.log(greetings);
// [ 'Hello', 'Guten tag', 'Salaam', 'Hola', 'Ciao' ]

console.log(sayHelloWorld(greetings));

/* [
  'Hello world',
  'Guten tag world',
  'Salaam world',
  'Hola world',
  'Ciao world'
] */

In this example, the map loops over each element in the greetings array and produce a new array called sayHelloWorld which has each greeting plus the word ‘world’ added to the end. But the original greetings array remains unchanged, as you can see by the console.log(greetings) line.

It can also be applied to complex arrays of objects, not just simple arrays.

And here is Array.map() creating a new array of objects:

const greetingsByCountry = [{language: "English", greeting: "Hello"},
                            {language: "German", greeting: "Guten tag"},
                            {language: "Arabic", greeting: "Salaam"},
                            {language: "Spanish", greeting: "Hola"},
                            {language: "Italian", greeting: "Ciao"}]

const helloByCountry = (hellos) => hellos.map(obj => {
    let newObj = {};
    newObj[obj.language] = obj.greeting;
    return newObj;
});

console.log(greetingsByCountry);
/* [
  { language: 'English', greeting: 'Hello' },
  { language: 'German', greeting: 'Guten tag' },
  { language: 'Arabic', greeting: 'Salaam' },
  { language: 'Spanish', greeting: 'Hola' },
  { language: 'Italian', greeting: 'Ciao' }
] */

// helloByCountry is the new array of reformatted objects
console.log(helloByCountry);
/* [
  { English: 'Hello' },
  { German: 'Guten tag' },
  { Arabic: 'Salaam' },
  { Spanish: 'Hola' },
  { Italian: 'Ciao' }
] */

This array of objects greetingsByCountry is used by map to create a new array of objects where the key is language and the value is greeting. The original array of objects remains unchanged though. map is very powerful and useful, and you'll find yourself reaching for it often.

Filter

Array.filter() is another useful method. It works by creating a new array from the original of all the elements in that array that pass a test implemented by the provided function. Let’s take a look at an example of this.

Check out how Array.filter() works:

const greetings = ['Hello', 'Guten tag', 'Salaam', 'Hola', 'Ciao'];

const filteredGreetings = filterParam => {
    return greetings.filter((hello) => 
    hello.toLowerCase().indexOf(filterParam.toLowerCase()) > -1
    )
};

// greetings is unchanged
console.log(greetings);
// [ 'Hello', 'Guten tag', 'Salaam', 'Hola', 'Ciao' ]

// filteredGreetings is a list of strings that contain the letter 'a'
console.log(filteredGreetings('a'));
// [ 'Guten tag', 'Salaam', 'Hola', 'Ciao' ]

The function filteredGreetings takes in a query parameter (the string of "a" in this case), and checks through the array of greetings to see if that letter exists in these words. If it does, it returns the first index in the string in which that letter exists, and adds the word to the new array filteredGreeting, and if it's not present in the string, the function returns -1, and it isn't added to the array.

Reduce

The Array.reduce() method is powerful: it can be used in a variety of ways ranging from simple to very complex, significantly cutting down on the amount of code needed to achieve results. In essence, reduce takes in an array, applies a function against something called an "accumulator" (more on that shortly) and each element in the array, and reduces it down to a single value.

Array.reduce() can also accept a second, optional, argument called an "initial value", which is simply the first value to use for the first callback of the function. If it’s not supplied, the first value in the array will be used instead.

Now an accumulator, as the name suggests, accumulates the value returned by the function each time the function’s callback is invoked. Makes sense, right? If the answer is "no", it’s totally fine, read on for a few examples to help clear things up.

Below is a simple example adding up numbers in an array using Array.reduce().

const prices = [5.15, 7.55, 3.82, 10.11, 6.79];

const sum = (nums) => nums.reduce((acc, currentVal) => {
    return acc + currentVal;
});

const sum2 = (nums) => nums.reduce((acc, currentVal, currentIndex) => {
    return acc + currentVal;
}, 10);

// the array of prices is still intact
console.log(prices);
// [ 5.15, 7.55, 3.82, 10.11, 6.79 ]

// sum is its own value
console.log(sum(prices));
// 33.42

// sum2 is its own value
console.log(sum2(prices));
// 43.42

Two examples using Array.reduce() on an array of prices.

Both functions take in the same array of numbers and run the same function (adding the accumulator to the current value being iterated over), but the second reduce function also takes in a second parameter of an initial value.

For the sum function, the array of nums is taken in, and since no initial value is provided as a second argument to reduce, the function begins at the index default of 1 (if not specified), and it iterates over each value adding the current index value to the accumulated value with each loop.

With sum2, an initial value val is provided as a second parameter to reduce so the function begins with that value as the first one passed to the callback, and then continues with all the other numbers in the array afterwards. This means that before the internal function even runs, the sum2’s value begins at 10 then loops through the rest of the array after that.

In the end, the prices array remains unchanged, and both sum and sum2 are their own individual values (numbers), as well.

Ready for another example? This one involves super heroes just for fun.

This second Array.reduce() example counts the number of times the various Avenger super hero names show up in the array.

const superHeroes = ['Black Widow', 'Iron Man', 'Hawk Eye', 'Hulk',
 'Thor', 'Iron Man', 'Thor', 'War Machine', 'Captain America'];

const heroCount = heroes =>  heroes.reduce((allHeroes, hero) => {
    if(hero in allHeroes) {
        allHeroes[hero]++;
    } else {
        allHeroes[hero] = 1;
    } 
    return allHeroes;
}, {});

// the array of Avengers is intact
console.log(superHeroes);
/* [
  'Black Widow',
  'Iron Man',
  'Hawk Eye',
  'Hulk',
  'Thor',
  'Iron Man',
  'Thor',
  'War Machine',
  'Captain America'
] */

// the number of times each Avenger is named is a rand new object
console.log(heroCount(superHeroes));
/* {
  'Black Widow': 1,
  'Iron Man': 2,
  'Hawk Eye': 1,
  Hulk: 1,
  Thor: 2,
  'War Machine': 1,
  'Captain America': 1
} */

In the above example, the superHeroes array is iterated over and if a hero is not included in the empty object passed as the second param to the reduce function, it creates an object with the key as that hero's name and the count set to 1. If the hero's name is already present in the object, it increments the number of times that name is seen in the string until the whole array's been accounted for.

This is another example of how reduce can be used to count the number of times a particular word is referenced in an array. The same kind of logic could also be used on a string, by simply splitting the string apart on spaces and then reducing through the resulting array and tallying up the words.

ForEach

The Array.forEach() method is a cleaner, more streamlined version of the OG for loop, but it is not always the right solution. For instance, if you need to loop over an array only until you find something that matches whatever argument it is you’re trying to find and then break out of the loop, forEach() is not the answer. forEach will keep going until it has gone through every element in that array. Period. Regardless of what it finds or doesn’t find.

If you want to stop a loop early when some value is found, an array method like some or every or find might better suit your needs.

Another big caveat of forEach is that it does not return anything. A forEach loop will always return a value of undefined. Unlike map or reduce, forEach is not chainable and does nothing more than run whatever function you've defined against each item in the array. A typical use case is to execute side effects at the end of a chain.

Here's an Array.forEach() example where we deduplicate a set of greetings.

const greetings = ['Hello', 'Guten tag', 'Salaam', 'Hello', 'Ciao', 'Hello'];

/* a new empty array to take in the modified strings */
const dedupedGreetings = new Set();

const sayHello = (hellos) => {
    hellos.forEach((hello) => {
        dedupedGreetings.add(hello);
    }
};

// greetings is unchanged
console.log(greetings);
// ['Hello', 'Guten tag', 'Salaam', 'Hello', 'Ciao']

// the new array has the unique greetings
console.log([...dedupedGreetings]);
// [ 'Hello', 'Guten tag', 'Salaam', 'Ciao' ]

The sayHello function loops over each string in the array and tried to add the hello to the dedupedGreetings Set. Since dedupedGreetings is declared outside of the forEach and we add values to it through our callback function, it works, but this is an example of a side effect. It’s not a perfect equivalent of a for loop, but it can certainly be useful in the right scenario.

Other Useful Array Methods

If there is a situation where you just need to know if an array includes a value, at least once, Array.includes(), Array.some() and Array.every() may be the methods you need.

Includes

Like the method name indicates, Array.includes() determines whether an array contains some element and returns a true or false boolean.

If the array includes it more than one time, it won’t indicate it because it will stop iterating as soon as it finds the first value that evaluates to true. includes can also take a second parameter as an argument: a search index, which specifies where the search of the array should begin according to the index.

Fun fact: includes was named as such because MooTools had already added a method to the array prototype called Array.contains which works slightly differently than how includes works.

To avoid breaking all the sites currently employing MooTools' contains method, Ecma International decided to rename their array method to includes instead.

Here’s a simple example of Array.includes() at work:

const greetings = ['Hello', 'Guten tag', 'Salaam', 'Hola', 'Ciao'];

const includesHello = (hellos) => {
    return hellos.includes('Hello');
}

// greetings is unchanged
console.log(greetings);
// ['Hello', 'Guten tag', 'Salaam', 'Hola', 'Ciao']

// returns `true` because `Hello` is in the greetings array
includesHello(greetings);
// true

This function just loops over the array and returns true if the string is contained anywhere within the array.

Some

Array.some() is similar to includes, but instead of just taking in an array, it takes in both an array and the value or function to evaluate the items in the array against.

some will iterate through an array and return true or false based on if it finds what it’s looking for. Also like includes, some will end its iteration as soon as it finds a value that evaluates to true.

Check out this example of Array.some():

const greetings = ['Hello', 'Guten tag', 'Salaam', 'Hola', 'Ciao'];

const someHello = (hellos, typeOfHello) => {
    return hellos.some(hello => hello === typeOfHello);
}

// greetings is unchanged
console.log(greetings);
// ['Hello', 'Guten tag', 'Salaam', 'Hola', 'Ciao'];

// the string `Hola` is present in the greetings array, so this function returns true
someHello(greetings, 'Hola');
// true

As you can see, the function someHello, takes two arguments: the array and the value the function is looking for (typeOfHello). Since ‘Hola’ exists, the function returns true. When you just need to know if a value is present at all in an array, some might be the function you need.

Every

Array.every() is like some, only every element in the array must pass the test and evaluate to true for the function to return true at the end. It’s pretty self-explanatory.

Here’s an example of Array.every() in action.

const greetings = ['Hello', 'Guten tag', 'Salaam', 'Hola', 'Ciao'];

const everyHelloGreaterThanFour = (hellos) => {
    return hellos.every(hello => hello.length >= 4);
}

const everyHelloGreaterThanSix = (hellos) => {
    return hellos.every(hello => hello.length >= 6);
}

// greetings is unchanged
console.log(greetings)
// ['Hello', 'Guten tag', 'Salaam', 'Hola', 'Ciao'];

// each greeting string has a length of 4 or more chars
everyHelloGreaterThanFour(greetings);
// true

// each greeting string is not 6 or more chars
everyHelloGreaterThanSiz(greetings);
// false

The first function everyHelloGreaterThanFour, checks the length of all the strings in the greetings array and since they’re all greater than or equal to 4, it returns true.

The second function everyHelloGreaterThanSix, does the same check on the greetings array, but with a length greater than or equal to 6, and it returns false since not all of the strings are at least that length.

Sort

Array.sort() is an incredibly useful method, but it is also tricky remembering how it sorts if a compare function is not supplied, which directs sort on how to compare and arrange the data.

Left to its own devices, sort will convert all elements to strings and compare each element according to its Unicode order (so although “Banana” comes before “cherry” as strings, in a numeric sort, 9 comes before 80, because numbers are converted to strings, “80” comes before “9” in Unicode order).

We fix this by passing simple functions to tell sort how to do its job on the values in encounters.

In general though, this is a very powerful and useful method that I use regularly, I just tend to supply a comparator function (I've been bitten many times when I'm lazy and don't).

NOTE: This method does modify the original array, but you’re only sorting it, and not directly altering the data, so there’s less chance of this type of change negatively impacting other blocks of code.

It can be computationally expensive though, if you're trying to sort a big enough array of comples objects.

Here’s Array.sort() on an array of numbers and on an array of objects.

const nums = [86, 10, 47, 29, 6, 55, 110];

// we tell the nums array how to sort its contents from lowest to highest
nums.sort((a, b) => a - b);

// nums is sorted, but otherwise unchanged
console.log(nums);
// [6, 10, 29, 47, 55, 86, 110]

const greetingsByCountry = [{language: "English", greeting: "Hello"},
                            {language: "German", greeting: "Guten tag"},
                            {language: "Arabic", greeting: "Salaam"},
                            {language: "Spanish", greeting: "Hola"},
                            {language: "Italian", greeting: "Ciao"}]

greetingsByCountry.sort((a, b) => {
    let langA = a.language.toUpperCase();
    let langB = b.language.toUpperCase();
    if(langA > langB){
        return 1;
    }
    if(langA < langB){
        return -1;
    }
    if(langA === langB){
        return 0;
    }
})

// now greetingsByCountry is sorted alphabetically by language
console.log(greetingsByCountry);
/* [
  { language: 'Arabic', greeting: 'Salaam' },
  { language: 'English', greeting: 'Hello' },
  { language: 'German', greeting: 'Guten tag' },
  { language: 'Italian', greeting: 'Ciao' },
  { language: 'Spanish', greeting: 'Hola' }
] */

The first function sorts the numbers array from low to high. The second function sorts the greetings by various countries by their language value, which first converts all the characters to uppercase, (lowercase would work too), to eliminate the differences in Unicode caused by upper or lower case letters.

Another nice thing to note is that many of these methods can be chained together, eliminating the need for extra variables, and making your code cleaner.

And that’s it. JavaScript arrays and their built in methods.

Conclusion

These are some of the methods I use most often when working with arrays in JavaScript. If you get good with these, you’ll be able manipulate code into doing exactly as you want in no time at all.

References & Further Resources

MDN documentation for Array.map()
MDN documentation for Array.filter()
MDN documentation for Array.reduce()
MDN documentation for Array.forEach()
MDN documentation for Array.includes()
MDN documentation for Array.some()
MDN documentation for Array.every()
MDN documentation for Array.sort()

How I Earned a Full-time Software Engineer Offer in 6 Months

Sun, 15 Apr 2018 00:00:00 GMT

</figure>

After I graduated from my 4 month coding bootcamp, I knew the hard work was only just beginning.

As I wrote about in my previous blog post, just under two years ago I made a decision to pursue the goal of becoming a web developer. I went to coding boot camp DigitalCrafts for 16 weeks, I was overwhelmed with new information on a daily basis, but I made it to end and graduated.

Not only did I graduate, but just the day before, I earned myself a contract-to-hire position as a Junior Java Developer at The Home Depot. The initial evaluation period was about three months, and, if I did well enough, and proved myself, I might get hired as a full-time Software Engineer.

Here’s some tips for how I approached this new challenge.

1. Aim High

Just as my goal had once been making it through coding boot camp, my new goal became getting a job offer — and that’s my first piece of advice:

Keep setting new goals to help propel you forward — keep reaching higher.

<figcaption>This view was my new goal (Software Engineer), and after that, perhaps that mountain in the distance, representing my next goal: Senior Software Engineer.</figcaption>

</figure>

On my first day at The Home Depot, and for many days after that, I am not ashamed to admit, I had next to no idea what was happening or what I was doing. Of course, I did my best to follow along with my developer pair of the day (The Home Depot subscribes to the pair programming model), but honestly, I don’t feel like I contributed much in my early days.

Once again, I was doing my best just to keep from drowning in all the new information being thrown at me: The Home Depot’s overall culture (welcoming and inclusive) and way of operating, my team’s way of developing software (TDD, agile and MVP-style with a focus on quality), and the tech stack my team used (which included all kinds of things I hadn’t encountered during boot camp, naturally: like Java, Jenkins and Pivotal Cloud Foundry, just to name a few).

2. Work, Work, Work, Work, Work

It was exciting and exhausting, and I was certain one day someone would catch on that I didn’t have it all figured out and I didn’t have a CS degree and they’d ask me to leave. Impostor syndrome is real, believe me — I still experience it pretty regularly.

But here’s my next piece of advice, which is a direct quote from my oldest brother, George:

“Work like hell for the first 6 months, listen carefully and learn from everyone you can — from the lowest of the low all the way up. Also, avoid the negative people, don’t engage in gossip and don’t consider any job beneath you if it takes the company forward.”

And he was right, even though I was continuously inundated with new material, new ways of doing things and learning a totally new programming language (Java), I needed to do all I could to get up to speed quickly and prove I could add value and was worthy of being hired as an associate.

<figcaption>I was never at the office quite this late, but there have been plenty of nights when I was one of the last people on the floor. I kind of like the peace and quiet actually - it’s easier to concentrate.</figcaption>

</figure>

Here’s some things that helped me do that:

3. Ask Questions & Write Down the Answers

I took notes — lots and lots of notes. And I asked questions whenever I could — even if it sounded stupid in my head, I swallowed my pride and asked anyway.

This is a segue, but one of the most pleasant surprises I encountered as I joined the tech community is that, by and large, people are really willing to help.

StackOverflow should have been a good indicator of this, but if you have a question and you’re on a Slack channel or with a bunch of other developers, they are almost always willing to share a similar experience they had and help you move forward or figure it out with you.

It’s really encouraging when you're struggling to know other people are in it with you and have your back.

Anyway, back to my point, I started with a notebook and a pen, then migrated it over to a note on my MacBook. That’s all it is — just a giant list of random notes, that I quickly learned would be invaluable to me.

Most of the notes are things that are simple enough to do in practice, it’s just remembering how to do them because I don’t need to do them very frequently (since then I've also been writing blog posts like these, and I can reference them too when I forget how to do things I've done before but it's been a while 😛).

For example,

setting up IntelliJ (my team’s IDE of choice) to run our SpringBoot Java microservices with environmental variables or Lombok annotations, or
knowing what database tables to query when troubleshooting production issues, or
having the commands handy for how to increase the logging levels in Pivotal Cloud Foundry applications for greater visibility into why network calls are failing.

<figcaption>No kidding, I (and most of my colleagues) have notebooks and digital note pads where we keep track of stuff that will come in handy in the future. Try it, you’ll be surprised how much you might reference it.</figcaption>

</figure>

None of these things are hard to do — but I do them infrequently enough, that it’s really helpful to be able to pull up my note sheet, and just search through it to get what I (or my teammates) need quickly.

4. Use the Resources Provided

My manager was kind enough to get me access to Pluralsight, a fantastic online learning platform, and put together a library of courses covering my team’s tech stack to help me know what to focus on as I ramped up.

He also gave me a piece of advice I’m still following to this day:

"Take an hour after work every day just to improve your skills: watch some videos, work on a side project, read a book or some blog posts. The point is to take time outside of our day to day to keep improving, and after a while, that ~5 hours of extra time a week will really add up, and help differentiate you from other devs who just work the 9–5."

I’ll admit, there have been plenty of days where I’m not motivated enough to do this by the time I get home, but I try to hit that 5 hour mark each week.

Sometimes it’s sitting down for a few hours on the weekend, sometimes it’s just staying late after work an hour or two and letting the traffic die down at the same time, sometimes it’s sneaking in a little bit of reading or coding during lunch or waiting for builds to finish at work, but I can say I certainly feel like my problem solving skills and coding ability have improved from this kind of continued after-hours learning.

And as I said above, I got to pair with a different dev on my team practically everyday, all of whom are knowledgeable, talented, capable people, so from them I was able to learn a whole lot.

I was introduced to lots of new ways of doing things: new ways of approaching problems, new ways of debugging solutions, new ways of managing application architecture and feature writing and testing, so not only could I do something one way, I knew several ways to do it by getting exposure to it from my different pairs.

I wasn’t afraid to ask questions either, I wanted to understand how we were building our applications and why. Besides combing through the code myself, which was sometimes such a mess of spaghetti that I’d end up very turned around and unsure of where to go next, I could ask my pair what this was doing or that was referencing or why the file structure was set up this way and get an answer I could understand and process.

The Slack channels for the entire Home Depot IT department are active and full of helpful, smart people I could also turn to when I needed assistance.

<figcaption>The resources I have access to helped me put the pieces of many puzzles together.</figcaption>

</figure>

5. Prepare to be Uncomfortable — If You’re Not Uncomfortable, You’re Not Growing

And here’s my final tip, and I think, what helped push me over the edge from "maybe we should hire her" to "let’s hire her":

Get used to being uncomfortable, get used to putting in extra time and effort, and show that you want to be there and you’re willing to persevere, even when it gets tough.

After about two months of working at The Home Depot, my manager sat me and another new developer who was hired a week before me down and gave us a personal project.

The goal of the project was to show we had learned enough about a new language (Java) to build a simple application using the same tech stacks our team used, but on our own. This meant an app with AngularJS and Node.js for the frontend, Java, JDBC, Springboot and Gradle for the backend, a MySQL database, and integration tests.

Ok, I can do this. I wasn’t sure how and I didn’t feel nearly as confident as I might sound, but I was ready to work. He gave us the specs and set us loose. Looking back, I can say without shame, it took me six weeks of after hours and weekend work, Pluralsight video tutorial references and walkthroughs, documentation references, Google and StackOverflow searches and little nudges of encouragement and question answering from my teammates to get there. When I look back at the code now, I cringe — it looks so hacky, but it worked.

<figcaption>Yes, sometimes I wanted to toss my laptop out the window, but I didn’t. I stuck it out.</figcaption>

</figure>

When both of us showed our manager what we accomplished he said: "Good job, it took you both way too long to get to this point. Now rebuild the backend service using Maven, XML and JPA/Hibernate." Three weeks (and again many after work hours later), we both had our second version to show him.

And here’s the kicker — the thing that I think showed him I could go above and beyond what he’d asked and helped prove I should have a permanent spot on his team.

To give a bit of background, first: our team uses Pivotal Cloud Foundry to host our apps. It’s a cloud platform like Google Cloud Platform or AWS or Microsoft’s Azure.

I was very unfamiliar with PCF and how it worked in most every way, but I went ahead and signed up for a free personal account while working on my proejct, PCF gave me some amount of free credit ($70, maybe?), and I pushed my applications (with quite a bit of difficulty since I didn’t really understand what a Java JAR or WAR file was) up to PCF and hosted it in their cloud with a database service attached.

When I finally showed my manager I’d built the app he’d asked me to, I could proudly do so with a link to the live site, not just my localhost site on my laptop. That, in my mind, is one of the reasons I think I was made an offer to join The Home Depot as a Software Engineer, just about a month later.

That project was by no means easy, but I learned so much from it (including that I learn best by doing, not by watching somebody else do it, but by physically opening an IDE and typing in the code — and fixing it when I make a typo and the console errors out), and I stuck it out, even though there were times I banged my head against the wall and wanted to just quit. I showed my grit and my desire to be there and succeed.

In the end, about 5 months after I began as a contract-to-hire with The Home Depot, I was rewarded with a full-time offer to become a Software Engineer, and I couldn’t have been more thrilled.

<figcaption>This is close to the amount of joy and accomplishment I was feeling upon receiving that full-time offer.</figcaption>

</figure>

In Conclusion, Just Keep at It

If you’re in your first developer job, or you’re still hunting for it, all I can say is: keep at it.

My advice in TL:DR form:

Aim for jobs asking for more experience than you have,
work your butt off — you’ll be amazed at the changes you see in yourself,
don’t feel dumb for asking questions, but write down the answers so you won’t have to ask a second time,
use any and all resources at your disposal (including your coworkers),
prepare to continue being uncomfortable — it means you’re doing it right and you’re growing.

It's cliché, but... if I could go from zero coding experience to a full time job offer and a title as a Software Engineer less than a year later, there’s no reason anyone else can’t too.

You got this.

</figure>

References & Further Resources

Definition of pair programming
Agile software development methodology
12 Factor App software development methodology

How I Went From a Digital Marketer to a Software Engineer in 4 Months

Sun, 08 Apr 2018 00:00:00 GMT

</figure>

Like more and more folks today, my journey to web development was not linear.

I don’t have the traditional CS background, I haven’t tinkered with computers since the age of 15, and if you’d told me 10 years ago when I graduated from high school that today, I’d be a Software Engineer for a Fortune 50 company, I would have looked back and asked what that even meant.

So what changed?

Well, to be perfectly honest, a lot. Almost three years ago now, I was working as a digital marketer for a small advertising agency in Atlanta. I did things like content strategy, social media strategy, branding and copy writing, and I liked it. The creativity, the challenge of making something from nothing, the variety of things to keep in mind.

What I didn’t like was the fact that we’d come up with incredible ideas for our clients — ideas that really pushed the envelope creatively, and then the clients either wouldn’t take our advice (in which case, why did they hire an advertising agency?), or they would approve our pitches and I would hand off my ideas to our teams of talented creatives and developers. I’d manage the project and the clients, and the graphic designers and developers would work their magic and come back with an amazing website.

And more than anything I wanted to know how they did it. How could they make something so cool with nothing but a computer and an idea? So besides asking questions whenever I could (and not really understanding the answers I got about things like Magento and APIs and landing pages), I started doing some self-learning on the basics of HTML, CSS and JavaScript with the help of CodeAcademy and Treehouse.

It opened up a whole new world to me, a world where I could build the ideas I came up with myself, but I had a hard time connecting the dots from the exercises I was doing online to seeing how that translated to a real, live website others could use. Since I was also working full-time then, I didn’t have much extra time to try and go further either.

Then, my world shifted. One Friday afternoon, my boss walked me down the street to a coffee shop and said without much preamble that that was my last day. I hadn’t done anything wrong, but they didn’t have enough marketing work to justify keeping me, so they were letting me go. I was stunned, and the following Monday, set adrift for the first time in my life, without a plan of what was happening next.

<figcaption>It wasn’t quite this dramatic, but it felt that way right after I was let go. I was in shock and hadn’t truly processed what had just happened.</figcaption>

</figure>

Taking a leap

For a few weeks, I struggled with deciding what I wanted to do next — pursue a new job in marketing, or go back to school for web development, and stop wondering for the rest of my life: what if I’d just tried it?

After some heartfelt conversations with my family and friends, my then fiancé (now husband) and a business acquaintance I’d made recently, but whose straightforward, no nonsense advice, immediately made me trust her, I decided to go for it.

I’d give this web development thing a fair shot, and go back to school for it. What did I have to lose? If I didn’t love it, at least I’d have new skills coding skills to add to my resume, which definitely couldn’t hurt in the marketing world.

I made the decision to go to one of the coding bootcamps in my city of Atlanta. The next thing to do was decide which one: there were three main bootcamps at the time, and after interviewing with each one, I chose DigitalCrafts. At the time, they were relatively new to the scene and I could tell by talking to the founders and their chief instructor, they were really invested in making sure their students succeeded. And for me personally, I wanted a program and curriculum to help me fast track this new career — in four months I’d either sink or swim. Period.

In July of 2016, I joined DigitalCrafts’ third cohort of full-stack web-development.

<figcaption>I had a plan. I wasn’t sure where I’d wind up at the end of all this, but I’d made a decision.</figcaption>

</figure>

The struggle was real

For 16 weeks, it felt like a fire hose of new information had been turned towards me, and my goal was to stay above water until the end.

This was, hands down, the hardest thing I’d ever done in my life. Five years prior, I graduated from Emory University’s Goizueta Business School, which I considered no small feat, but it paled in comparison to the demanding pace and curriculum at DC.

To put it in perspective: our cohort began with 15 students, by week 8 (halfway through), we were down to 12.

During that time we learned so much: HTML, CSS, JavaScript, jQuery, AngularJS, NodeJS, Express, Python, Flask, Socket.io, MySQL, MongoDB, Git, Github the list goes on, but more than that, we learned how to learn and think like developers.

I know it sounds cliché, but learning to break down problems (and projects) into smaller, more manageable chunks, how to ask the right questions to overcome obstacles, and how to persevere, even when it seemed like I could learn no more or figure out why my code wasn’t doing what I wanted.

<figcaption>This is specific to the framework of AngularJS, but it’s accurate for how I felt going through boot camp: awesome highs when I learned something new and got it to work, mingled with lows as new material was tossed at me and I hadn’t figured it out yet.</figcaption>

I won’t lie, there were days I came home after 10 hours at school and cried in my husband’s arms, feeling like an impostor, feeling like maybe I wasn’t cut out to be a developer, feeling like I couldn’t do it, but day after day, I went back and I kept trying.

This is critical: just keep trying. Fail and try again. Trust me.

<figcaption>A reenactment of many, many moments during bootcamp. This guy is all of us many times when programming.</figcaption>

</figure>

Preparation is key

If there’s one thing DigitalCrafts taught me well, it was the importance of my story and my digital presences.

Most of my classmates and I had next to no prior development experience, so telling my story of how I’d come to web development and why I deserved to be given the time of day was paramount. This included things like:

making sure my resume highlighted important qualities such as my ability to learn new technologies and concepts quickly,
showcasing projects that demonstrated my ability to build working applications with a variety of development languages on my portfolio site,
linking up my Github and keeping the code clean for interested parties to see my skills, and
practicing coding and whiteboard interviewing.

All so I’d be ready, when it came time for the real thing.

I spent hours poring over my code, practicing coding challenges, reworking my resume and tweaking my portfolio until it was just perfect (on both mobile and desktop, of course — thank you, Bootstrap). And then came the nerve-wracking part, actually applying for web development jobs.

<figcaption>The hallmarks of programming: whiteboards, dry erase markers and tons of pseudo-code and diagrams. Practice at this kind of stuff makes perfect (and helps you prepare for when the real test comes).</figcaption>

</figure>

Making opportunities happen

About three-quarters of the way through my bootcamp, I was contacted by a tech recruiter, and we connected really well.

I told him I was looking for a company where I could have a positive impact and work with really good developers, people more experienced than myself, people I could learn from - basically the people I wanted to become one day.

One of my biggest takeaways from my coding bootcamp was just how much I didn’t know, and my desire to improve and keep learning new things helped me decide, at least initially, that a great way to do that would be with lots of talented devs to learn from.

He had me send him my resume, and I dove back into learning as much as I could during my bootcamp.

In the meantime, I’d been applying to any and all junior development jobs I came across on my own, as well. I had daily job alerts sent to my inbox from Dice, Indeed, LinkedIn, Glassdoor, I was on the Atlanta tech Slack channel Tech404, and I was attending local meet ups like the Atlanta JavaScript Meetup and the PyAtl: Altanta Python Programmers Meetup to meet more people in my local tech community.

From past experience, I knew networking would be key to getting a job — the more people I had a chance to meet, the more I could show them I might be just the excited new developer they needed for their team.

<figcaption>Meetups are a great way to meet some really cool, super smart people in the tech industry. </figcaption>

</figure>

To stay organized, I kept a spreadsheet of all the jobs I’d applied for, what date I’d applied on, if I’d received a response (both yeses and nos), contact details (if I could get them), any other extra info worth remembering, and links to the job posting.

I also had a folder full of customized cover letters I included with all my job applications to show my interest, why I’d be an excellent fit for the job and how I could contribute on day one.

The call

Two weeks before I was due to graduate, the recruiter I’d spoken to before called me back.

He’d put my resume in front of a hiring manager at The Home Depot and wanted to set up a phone screen. I was nervous, but I said yes and a few days later we talked. I don’t remember exactly what he asked me — looking back, we discussed pretty simple technical questions about things like databases and JavaScript, projects I’d built and obstacles I’d overcome during the process, and behavioral questions about how I’d handle different situations with coworkers. Overall, I felt ok after the call ended. When the recruiter called me back later that day, he told me whatever I’d said, I’d knocked it out of the park, they wanted to bring me in for an in-person interview!

The day before I graduated, I went in for my in-person interview. I was nervous and didn’t know what to expect. It lasted four hours. The first hour and a half was with the hiring manager I’d spoken to on the phone, we did some simple white boarding questions, he asked me some more questions about how I would approach different challenges (both technical and behavioral), then I was paired up with one of the senior developers on his team and we worked on solving a problem for the application the team was building.

Luckily for me, the application’s UI was being built using AngularJS, which I’d learned in bootcamp, so I was able to figure it out (with some help from my pair). At the end of the interview he showed me the code he’d written before I arrived to solve the same problem and compared it to my code, and they were almost identical! Wow, that felt good.

<figcaption>I didn’t feel nearly as cool and collected as these two look. I was trying not to freeze up and just make the code work.</figcaption>

</figure>

I walked out just after noon feeling pretty good about my interview performance and very excited about how the company viewed development and improving its current technologies.

Here’s a tip I learned even before I graduated from college about job hunting: during the interview, I get the name and email address of any one I speak to.

That way, right after the interview, I can send a thank you email reiterating how much I appreciated the time and consideration, touching on something we discussed and saying how much I’d like to be part of their team (as long as it’s really how I feel).

It may sound like overkill, but I also make a point to send a handwritten thank you note — they’re so rare nowadays and I know it can make a difference (people I’ve been hired by in the past have told me so, after the fact).

Bottom line: show them you want to be there, the right attitude and eagerness about something definitely makes a difference.

What happened

In this particular case, I didn’t even have time to send anything beyond thank you emails, that same afternoon I got a call from my recruiter that The Home Depot wanted to extend me an offer to join as a Junior Java Developer. And I was thrilled to accept it (I also sent handwritten thank you notes too, saying how excited I was for the opportunity they were giving me).

<figcaption>The view wasn’t this great, but this is exactly how I felt when I got the offer.</figcaption>

</figure>

And from there, I began the next hardest thing I’d ever done, growing into my first developer role. But that’s for another post.

The final stats

To give a quick breakdown of my journey, here’s what I did by the numbers.

I joined DigitalCrafts for a 4 month, immersive bootcamp, covering both front and backend technologies. This was about 680 hours of in-classroom experience and an untold amount of time after class and on weekends when I was fixing projects, polishing up my website and doing whatever I could to learn more.

In the space of about 40 days before I graduated, I applied for 53 jobs, received 10 email rejections, conducted 6 phone interviews, went in for 3 in-person interviews and was extended 1 full-time job offer.

Just 16 short weeks ago, I’d known next to nothing about how web development worked, and by the end I had full-stack projects and a full-time development job offer.

I am extremely lucky to have a supportive spouse and family, who made it possible for me to fully dedicate myself to making this transition to web development happen, but believe me, if I can do it, you can too.

Now get to coding!

</figure>

Paige Niedringhaus RSS feed

Getting the Most out of Claude Code

Introduction

Claude Code

Context matters: give Claude plenty of it

Use Claude's planning capabilities

Help Claude help itself with MCP tools

Lean on Claude's expertise in unfamiliar territory

Don't trust Claude: review and verify

Conclusion

References & Further Resources

Run Multiple npm Publishing Scripts with Trusted Publishing (OIDC) via GitHub Reusable Workflows

Introduction

GitHub Actions reusable workflows

Create a publish-switch.yml file in the .github/workflows/ folder

Configure the npm repo's Trusted Publisher to point towards the publish-switch.yml file

Voilà! OIDC trusted publishing to npm

Conclusion

References & Further Resources

Reviewing the BenQ RD Series Monitor for Developers

Introduction

Initial assembly and setup of the RD280U was a breeze

Noteworthy RD280U features for devs

"Coding Mode"

Additional monitor features

My favorite monitor feature (I didn't see this one coming tbh)

Conclusion

References & Further Resources

Persist Zoom and Bounds in a React Plotly.js Map

Introduction

Install Plotly.js and React-Plotly.js

Create a Reusable Map Component

Leverage React's useState() to track map state and visible devices in the map container

Interact with the map component and verify the coordinates and device count persist

Conclusion

References & Further Resources

How to Cross Link Plotly.js Charts in a React App

Introduction

Install Plotly.js and React-Plotly.js

Create a Reusable Chart Component

Leverage React's useState() to track date selections in the charts

Test out the cross linked charts

Conclusion

References & Further Resources

Build a Custom Time Slider Component with Ant Design and Next.js

Introduction

Install Ant Design and Day.js

Ant Design Slider

Day.js

Create a TimeSlider component

Calculate and style the tick marks to handle different date ranges appropriately

Format the tooltip

Style the track and marks to handle current and future dates differently

Import the TimeSlider component into the parent page

Test out the Time Slider component

Conclusion

References & Further Resources

Automatically Publish a Repo as a PyPI Library with GitHub Actions

Introduction

Notehub Py

Create a scripts.py and config.json file to automate generating the library and packaging it for distro to PyPI

Set up a PyPI account and library to publish to

Write a GitHub Actions workflow to publish to PyPI

Conclusion

References & Further Resources

Animate an Auto-Scrolling Carousel with Only HTML and CSS

Introduction

CSS keyframes and scroll snap

CSS keyframes

CSS scroll snap

Set up the HTML and CSS

Conclusion

References & Further Resources

Lean on CSS Clip Path to Make Cool Shapes in the DOM without Images

Introduction

CSS clip-path

The polygon I needed to recreate with CSS

Set up the HTML and CSS

Add the CSS clip-path to shape puzzle-section

Conclusion

Create a `publish-switch.yml` file in the `.github/workflows/` folder

Configure the npm repo's Trusted Publisher to point towards the `publish-switch.yml` file

Create a `scripts.py` and `config.json` file to automate generating the library and packaging it for distro to PyPI

Create a GitHub Actions workflow to run when the `openapi.yaml` file is updated

Write one shell script to copy the `openapi.yaml` file to the Notehub JS repo

Configure the root `package.json` to point to the subfolder where the Notehub JS library code lives