Frontside Software RSS Feed

Announcing Effection 3.0 -- Structured Concurrency and Effects for JavaScript

Charles Lowell — Mon, 18 Dec 2023 20:00:00 GMT

Effection is Structured Concurrency and Effects for JavaScript.

It’s a library purpose-built to help developers write code that can easily manage the most complex concurrent computations whether in the frontend or backend. It provides the guarantees of Structured Concurrency in order to make code leak-proof by default, and it achieves this while being simple to read, understand, and maintain.

Just over five years ago, in November of 2018, when Structured Concurrency was a little-known peculiarity, we started a helper library to coordinate the many moving parts in some of our open-source projects (such as Interactors). Since then, it has grown into a mature project in its own right; deployed in production worldwide, helping developers use Structured Concurrency to push the limit of what is possible with JavaScript.

Today, we are pushing things further and concentrating even more power into the hands of JavaScript developers with the release of Effection 3.0. This new version refines and simplifies the abstractions we first created in version 2.0 while introducing some new ones of exceptional power. This release has much to look forward to, including the new Context API and an even better TypeScript experience. Most of all, however, we’re proud of how we’ve realigned all of the APIs so that working with Effection feels just like working with JavaScript, only with the protections of structured concurrency baked in.

That’s why we say more than ever before: Effection is Structured Concurrency and Effects for JavaScript

Here’s an overview of some key features you’ll find in Effection 3.0

- Excellent TypeScript Experience: TypeScript types have been carefully re-imagined to be as small and helpful as possible.

- Embrace of JavaScript: Version 3 is easier to learn and safer to integrate into your projects than ever because of its dedication to following JavaScript standards in spirit and form.

- Context API: Shared environmental data (like auth tokens), shared resources, and contextual (aka algebraic) effect handlers have at least one thing in common: They are all made possible by the new Context API.

- Rebuilt with Delimited Continuations: The v3 API is so simple it feels like it almost isn’t there. But that doesn’t mean there isn’t some serious brainpower behind the ideas that make this possible.

🏷️ Excellent TypeScript Experience

Since version 2.0, Effection itself has been written in TypeScript. As a result, a lot of thought was put into making it pleasant to use there. However, some rough edges remained.

The first area where the TypeScript experience improved is in the Operation<T> type itself. In the 2.x versions of Effection, it was a union of seven member types, one of which was undefined. It was unclear what you should set as your return type if you were writing a function that returned an Operation. Compound this with the fact that all Effection APIs were inconsistent in what they returned. Some returned Resource<T>, others OperationFunction<T>. Now, there is a single Operation<T>, which is a simple interface that can be used consistently as a function parameter and return type.

However, the most vexing issue with Effection v2 in TypeScript arose because the generator type prevented the left side of a yield expression from being inferred properly. That meant you had to annotate values bound in this way manually. For example, if we had an operation of type createNumber(): Operation<number>, then we would have to manually tell TypeScript that the result of createNumber() was, in fact, a number, even though the return type already clearly indicated this.

function* operation() {
  // we have to declare that `count` is a number
  let count: number = yield createNumber();
  return count.toExponential();
}

But that was only ever a hint, not a constraint, and a completely bogus annotation would breeze right through the type checker as well.

function* operation() {
  // we have to declare that `count` is a number
  let count: number = yield createString();
  return count.toExponential() // TypeError!
}

This was a major Achilles heel for TypeScript developers who wanted to use Effection. They expected the complete experience, not just of compile-time type checking, but also all the language server tooling that having type information enables. Sadly, the TypeScript experience with Effection v2 fell short of what they expected.

Effection 3.0 makes this completely a non-issue by using yield* to evaluate operations in the context of another. This has many benefits, including shorter stack traces and lower memory overhead, but one of the greatest is that it completely solves the type inference problem. This is because the type of a yield* expression is independent of the type of function that evaluates it. Now, if we try to assign an incompatible type from an operation, we’ll get a type-checking error:

function* operation() {
  //@ts-expect-error `string` is not assignable to `number`!
  let count = yield* createString();
  return count.toExponential()
}

✨ New Context API

Sometimes, you need to share something across a wide range of operations. It could be anything from a simple string like an auth token to a shared service that other code needs to function. But so much depends on this shared thing that you don’t want to pollute your APIs by passing it around as an argument everywhere. Instead, you want it to be “just there” as part of the environment in the same way the document reference is “just there” on a web page.

Effection now ships with an API that solves this need elegantly. It is deceptively simple, but don’t let that fool you. The new Context API is jam-packed with power, and it’s one of the features we’re most thrilled to be releasing.

To see this in action, let’s make a Username context to track our current user so that we can greet them from an operation.

import { createContext } from "effection";

// create the username context;
const Username = createContext(`username`);

We can now use this context from within any operation! Let’s create an operation to greet the current user:

// greet the current user, whoever that is
export function* sayHello() {
  let username = yield* Username;
  console.log(`Hello ${username}!`)
}

await main(function*() {
  yield* Username.set("anonymous");
  yield* sayHello();
})

If we run the above code, it will print the following to the console.

Hello anonymous!

But here is the kicker. Context is sensitive to scope.

Because of this, when you set the value of a context, it is only visible to that scope and its children. All other scopes are completely unaffected.

await main(function*() {
  yield* Username.set("anonymous");
  yield* sayHello();

  yield* call(function*() {
    yield* Username.set("bob");
    yield* sayHello();
  });

  yield* call(function*() {
    yield* Username.set("alice");
    yield* sayHello();
  });

  yield* sayHello();
})

prints

Hello anonymous!
Hello bob!
Hello alice!
Hello anonymous!

As you can see, the child operations get their own private definition of Username that neither interferes with each other, nor the definition of their parent.

Call it a foundation for dependency injection, or a basis for algebraic effect handlers, the range of uses for this mechanism are massive and we can’t wait to see what folks end up doing with it.

❤️ Embrace the JavaScript

Our goal with Effection has always been to make it easy to write JavaScript code protected by Structured Concurrency's guardrails. That’s why our project’s tagline is “Structured Concurrency and Effects for JavaScript.” We believe you shouldn’t have to learn an entirely new way to write programs to achieve the benefits of Structured Concurrency. Instead, we believe that the only new thing you should have to learn is Structured Concurrency, and everything else should feel like plain JavaScript/TypeScript every step of the way.

With version 3.0, we’ve doubled down on this idea by re-imagining our APIs as mirrors to existing counterparts in JavaScript. That way, if you know how to do it in JavaScript, you know how to do it in Effection. We call this conversion between vanilla JavaScript and Effection our “Async Rosetta Stone.”

Async	Effection
Promise	Operation
new Promise()	action()
await	yield*
async function	function*
AsyncIterable	Stream
AsyncIterator	Subscription
for await	for yield*

There is a structured analog for any construct you use for async programming. For everything else, JavaScript and its wider ecosystem is your standard library.

Consider this example from MDN

Async

function resolveAfter2Seconds(x) {
  return new Promise((resolve) => {
    setTimeout(() => resolve(x), 2000);
  });
}

// async function expression assigned to a variable
const add = async function (x) {
  const a = await resolveAfter2Seconds(20);
  const b = await resolveAfter2Seconds(30);
  return x + a + b;
};

Effection

function resolveAfter2Seconds(x) {
  return action((resolve) => {
    let timeout = setTimeout(() => resolve(x), 2000);
	  return () => clearTimeout(timeout);
  });
}

// operation function expression assigned to a variable
const add = function* (x) {
  const a = yield* resolveAfter2Seconds(20);
  const b = yield* resolveAfter2Seconds(30);
  return x + a + b;
};

Interoperation between these two worlds will always be our project’s foremost concern. For example, creating a Stream at runtime is trivial from a reference to an AsyncIterable and vice-versa. The same is true for Subscription and AsyncIterator

Effection is dedicated to keeping this alignment with core JavaScript in place both now and into the future so that developers can have peace of mind knowing that Effection will always be:

Easy to learn for any developer encountering Effection code for the first time. If they know JavaScript, they (mostly) know Effection.
Easy to adapt to any existing codebase, no matter what other paradigms it may use.
Easy to “sprinkle” into a codebase without requiring massive refactors.

🧠 Rebuilt with Delimited Continuations

You might think that a robust implementation of Structured Concurrency and Effects for JavaScript could easily be large, complex, difficult to understand, and a pain to debug. And if you were talking about Effection prior to version 3.0, you might even be right!

Versions 1.x and 2.x used a combination of state machines and cascading chains of callbacks that fired events and transitioned those state machines to orchestrate code to run at the right time. In a way, those past versions of Effection jumped on the grenade of callback hell so that users didn’t have to.

While that was a worthy tradeoff, Effection 3.0 gets to have its cake and eat it, too by undergirding its entire implementation on the foundation of delimited continuations. These curious constructs from the world of functional programming are extraordinarily powerful artifacts. They can express all the standard control mechanisms we’re familiar with, from while loops to try/catchblocks, to algebraic effect handlers.

Having an abstraction of this power drive Effection v3 means that its core logic can be expressed in only about 150 lines of TypeScript. And because it isn’t based on asynchronous callbacks but instead delimited continuations, execution is linear and straightforwardly follows its source code. As a result, it is an order of magnitude easier to understand and maintain.

We’re packing a more powerful, performant, and maintainable library into a much smaller footprint thanks to delimited continuations. The bundle size has been slashed in half and clocks in at a miniscule 4.6kb gzipped.

To delve deeper into the subject of delimited continuations in JavaScript, we recommend:

- Eric Bower’s talk for Michigan TypeScript

- On Wikipedia

- TypeScript tutorial on delimited continuations

Get Started With Effection 3.0 Today!

Wondering what the next step is? Whether you're beginning a new project, or wanting to bring more clarity to an existing one, Effection is meant to be both the simplest and the easiest way to use structured concurrency and effects in JavaScript.

Check it out on GitHub, have a look at our guide to getting started, or drop into our Discord Server to say hi. We're always eager to talk, and look forward to hearing from you all soon.

The await event horizon in JavaScript

Charles Lowell — Mon, 11 Dec 2023 20:00:00 GMT

There is a boundary around every black hole where the velocity required to escape its gravitational pull exceeds the speed of light. Once anything, including light itself, passes over that threshold, it is trapped inside the mysterious interior of the black hole forever. There is no escape, and there is no return back to the rest of the universe. This boundary is called the black hole’s event horizon.

A similar boundary exists around every JavaScript Promise, and once the flow of execution crosses over it, there is no way to forcibly escape and return back from whence it came. I refer to this boundary as the Promise’s await event horizon.

An async function traverses an await event horizon every time it pauses for the result of a Promise. When it does, control can never return back to it until such time as that Promise settles; and there is absolutely no way to guarantee that it (or any) Promise will ever settle. It might happen in the next tick of the event loop, or in the worst case scenario, it just might never settle at all. When that happens, the poor waiting function is stuck forever helpless… suspended like an insect in amber.

This is not theoretical. It is of important, practical concern. For example, consider this async function that implements the very common pattern of doing some setup, performing an operation, and then doing necessary teardown. In the code below, the async function acquires a lock, awaits a function it receives as an argument, and then, once that operation is complete, releases the lock.

async function protect(work) {
  let lock = await acquireLock();
  try {
    await work();
  } finally {
    release(lock);
  }
}

But what happens if the Promise returned by work() never settles? The answer is that the protect() function has passed over the await event horizon, never to resume. As a result, the lock it acquired at its start is never released, and so we say the lock has been “leaked”.

Resource leaks are considered by many to be the most insidious category of bug in software because of the difficulty in tracking them down coupled with the fact that they often lay hidden until the system is under the stress of a heavy workload.

Of course as a rule, most promises do settle if we let them run long enough and so for the most part we abide by the blissful convention that they will. But happy assumptions rarely pan out at scale. In fact, we find out pretty quickly that it isn’t just promises taking forever that are problematic.

Let’s suppose that we wanted to call our protect() function from within a command line interface, and that when the user hits CTRL-C, we exit our process. If our work takes ten seconds, but the user hits CTRL-C after nine and a half seconds, then control never returns from beyond the await event horizon and the lock is leaked once again.

We aren’t talking about a promise that never settles here. We’re talking about one that would have settled imminently, and yet it only took a difference of five hundred milliseconds for the leak to manifest.

Is Explicit Resource Management a solution?

In a word: no.

Explicit resource management, a stage 3 TC39 proposal as of this writing, allows you to bundle setup and teardown code together. It saves you from going through the ceremony of a try/catch block and makes it much more difficult to inadvertantly leak resources. If our hypothetical locking mechanism had it built in, it would allow us to write our protect() function much more clearly.

export async function protect(work) {
  using lock = await acquireLock();
  await work();
}

When protect() finishes and lock passes out of scope, it is automatically released according to its pre-bundled deallocation logic.

While this a handy improvement, it does nothing to change the fundamental physics of the await event horizon. Once flow control passes through it, there is no coming back until work() settles. If that takes too long, then protect() will take too long, and as a result, the automatic destruction of the lock resource will never be triggered. In other words, it is leaked.

Does AbortSignal help?

In theory yes, but in actual practice kinda-not-really. Abort signals can, when applied with discipline, remediate the problem somewhat, but they cannot solve it. It begins with the reality that there is no agreed upon way to “cancel” a promise when handed an AbortSignal. In fact, it is a very thorny issue which caused T39 to throw up its hands almost a decade ago, and if you think you know the answer offhand, then the odds are that you haven’t thought about it long enough. However if we must, one way is to wrap every promise in a safe() function that serves as a barrier to protect us from the await event horizon.

export function safe(promise, signal) {
  return Promise.race([
    promise,
    new Promise((_,reject) => signal.addEventListener("abort", reject)),
  ])
}

If signal fires before promise settles, then our safe function will immediately reject, returning control to the caller by throwing an error. With this mechanism in hand, we can re-write our protect() function to thread an abort signal throughout the entire computation.

export async function protect(work, signal) {
  let lock = await safe(acquireLock(), signal);

  try {
    await safe(work(signal), signal);
  } finally {
    release(lock);
  }
}

The function now uses safe() to wrap every await expression in a circle of protection that prevents execution from becoming trapped beyond its event horizon. In addition, it passes signal down to work() so that it, and any functions that it calls can do the same. Now, when signal is fired, no matter if work() becomes stuck, our protect() function will exit, and the lock will be released. However, the approach still comes with this serious and unavoidable caveat: the abort signal is a hope, not a constraint.

It’s not just that an extra abort signal is cumbersome to both use and pass around. It is. It’s that if work() or any of the functions that it calls, or any of the functions that they call, fail to use signal and safe() then we are right back in the same boat of having async functions in our call tree that become trapped beyond the await event horizon and leak resources as a result.

What’s fundamentally missing is the power of abstraction: the freedom to think about work() as a black box and feel secure that it will return control to its caller whenever it needs it most, no matter how work() is constructed internally. Using abort signals, the only way to achieve this is would be to read its source code and the source code of all its transitive dependencies to ensure that they also observe abort signal discipline. In practice, nobody would do that, and the scarcity of libraries that actually integrate abort signals shows that nobody does. While they can work around the obstacle through the imposition of discipline, they cannot make it disappear altogether. The fact remains that once control passes through the await event horizon, it cannot be brought back.

Structured Concurrency and the await event horizon

There are many remediations for the problem this poses, but most of them amount to some variation of the abort signal. The await event horizon however, remains axiomatic because the mechanics of it are baked into the heart of the runtime. There is no way to reliably pierce through it.

The consequence of this is that the minimum lifetime of any given async function is determined solely by the lifetime of its innermost promises. In other words, the natural lifetime of an async function is determined from the inside out.

protect() cannot continue until work() settles… whenever that may be. So it is work() that determines when protect() can continue, and it is protect() that determines the natural lifetime of our main function. Anything that overruns the exit of our process is in the “danger zone” of being leaked. This includes our hypothetical lock

Any code that runs longer than needed is in danger of being leaked

In fact, this is precisely the opposite of what is required by a Structurally Concurrent system. Namely, that the maximum lifetime of a function is constrained by the lifetime of the function that calls it. Instead of waiting around for async operations to complete that have no bearing on the outcome of a computation, a structurally concurrent system will return from those functions immediately the moment they are no longer necessary. In our example of the command line interface, as soon as the user hits ctrl-c, everything else becomes immediately irrelevant.

What we would like to see in this case is the forcible return of control back to the protect() function, so that it can run its finally block so that the lock is not leaked and the process exits gracefully.

A well behaved operation always returns

However, in order to enforce the shutdown of these irrelevant functions, there must be some mechanism by which to impose a return of control from the top down. But we’ve just seen how in async functions, once control passes through the await event horizon, it cannot be brought back. It’s for this reason that primitives based on async functions can at best hope for structured concurrency, but they can never guarantee it.

Structured Concurrency and JavaScript

You might be thinking at this point that structured concurrency in JavaScript is a lost cause because it can’t be achieved with async functions. Far from it! Structured Concurrency is not only possible, you can already find it out in the wild today in the likes of projects like Effection, Effect-TS, and StarFX (to name a few). These libraries come in all shapes and sizes, but one thing that they all share in common is an embrace of generator functions as a core technique.

This is because generators functions in JavaScript are not limited by the await event horizon. They represent full-fledged delimited continuations, which is one of the most (if not the most) powerful flow control primitive there is. Without delving into what delimited continuations are, suffice it to say that they can be used to express any other control mechanism you’d care to implement, from while loops, to try/catch blocks to algebraic effect handlers. In fact, in their essence, async functions themselves are just a watered down version of generator functions limited to the specific domain of promise handling.

Critically for our use-case, generator functions allow for an explicit return() which signals them to exit posthaste from wherever they are currently executing. Yet as they do, they will still follow critical code paths such as finally {} blocks or explicit resource methods.

Perhaps someday in the distant future, async functions will provide the programmer with a mechanism to escape the await event horizon, but until that time, a structured concurrency model based on async/await will be nothing more than science fiction.

Backstage: Unblock Innersource Development with Plugins and Code Owners.

Taras Mankovski — Tue, 24 Oct 2023 05:00:00 GMT

Backstage empowers platform and product teams within their organization to extend their internal developer portal in an inner-source manner. Keeping up with the growth of an inner-source Backstage portal brings its challenges. In this blog post, we’ll describe how successful Backstage adopters use plugins and the concept of code owners to unblock the development of their Backstage portal.

Most successful Backstage adopters have a single team responsible for the overall Backstage portal - we’ll call this team the Backstage team. The default project structure for a Backstage portal is a monorepo, meaning that a single repository contains multiple packages. The Backstage team is responsible for the Backstage repository. Developers from other teams make inner-source contributions to the Backstage repository by making pull or merge requests. The Backstage team is responsible for reviewing inner-source contributions to ensure they follow Backstage best practices.

As the project grows and the number of inner-source contributions increases, so does the number of code reviews that the Backstage team has to review. At some point in every Backstage project, the inner-source contributions exceed the bandwidth the Backstage team has to perform requested code reviews. This situation can be difficult for everyone involved. The Backstage team is responsible for the overall code health of the project - not reviewing code is a bad idea. On the other hand, teams making inner-source contributions have their commitments and timelines that they need to meet.

Most successful Backstage adopters resolve this conflict by giving inner-source contributors autonomy to review their code while requiring the Backstage team’s review of shared components. Enforcing these boundaries is done by defining areas of ownership within the Backstage repository using code owners. Code owners are configured via a file in a repository that contains a list of directories and teams responsible for the content of each directory. The team responsible for a directory must review a pull request that changes a file in their directory. GitHub, GitLab, and BitBucket support code owners. Here is an example of CODEOWNERS file of the Backstage project itself.

One of the challenges of adopting code owners is that not every Backstage portal is easy to describe in a CODEOWNERS file. This usually happens because the code in the repository is not organized into Backstage plugins. You can often tell if this is happening if your project’s packages/app has lots of card components. Some of these card components should be in a directory of the plugin owned by the team that’s maintaining that plugin. For example, a team responsible for CI/CD may have a CI/CD card in the Backstage app instead of importing that component from their plugin in the repository. Refactoring the project to create clear boundaries between the Backstage team’s code and the inner-source contributor’s code is often the first step to introducing code owners to reduce the code review burden on the Backstage team.

In this blog post, we covered how to use code owners and Backstage plugins to streamline inner-source contribution by giving individual teams the autonomy to review their plugins without giving up the oversight necessary for the Backstage team to support the portal’s development. We hope this blog post is helpful to you on your organization’s journey to a successful Backstage adoption.

If you'd like to learn more about building Backstage plugins, check out our workshop on Advanced Backstage Plugin Development. If you’d like Frontside developers to work alongside your team to make your Backstage portal successful, check out our Backstage consulting services.

Backstage: How to fix linting errors without slowing down development

Min Kim — Mon, 16 Oct 2023 05:00:00 GMT

TL;DR This blog post outlines the steps in creating a patch for the lint command in Backstage, which organizes errors by package, rules, and overall counts. The author explains how to use the generated lint error data to monitor and manage lint issues efficiently, and emphasizes the importance of tracking progress in reducing lint errors.

I recently worked with a team that had the lint command in their Backstage project misconfigured. The result? They ended up amassing an overwhelming number of lint errors - literally hundreds upon hundreds. In fact, the report output by the lint command was so long it would max out the terminal.

Normally, you would run the lint command in CI for every commit of your pull requests. This practice ensures that any new lint errors introduced by the changes would prevent the pull request from being merged. However, we could not set up this configuration right away; doing so would have brought development to a screeching halt.

With so many lint errors, it was difficult to pinpoint a starting point. To gain a clearer overview of the types and quantity of lint errors we were dealing with, I made modifications to the Backstage lint command. The modified command produced reports within a structured file system and generated an errors' summary.

Writing the Patch

The lint command of the backstage-cli package can be found here. We're going to write a patch for it to produce the summary file so let's start by installing patch-package:

yarn add patch-package -W

And modify the root package.json of your Backstage project:

"scripts": {
+  "postinstall": "patch-package"
}

Then navigate to the correct lint command file in ./node_modules/@backstage/cli/dist/cjs/lint. There will be three Javascript files for linting - one for the base lint command, another for the repo lint command, and a third for the versions lint command. The file we're going to modify is the one designated for the repo lint command.

Here is a high level view of the patch we're going to be writing:

create `errors` directory
create `summary` file

for `errors` of each `package`
  - create directory for this specific package
  - output lint report of this package to a text file in its corresponding directory
  - output the total lint error count of this package and list the rules that are being violated

calculate grand total count of lint errors for the entire project
combine the total count of lint errors by rule for the entire project

And here's the actual patch. Typically we would use diff syntax to indicate which lines to be added, but in order to make the code easier to read, we will display the code snippet with Javascript syntax highlighting.

The code snippet below is quite long so you can check out a working demo of a Backstage app with the lint patch here. The actual patch can be found here, and you could also follow along its commit history for a more step-by-step approach in creating the patch.

var fs = require('fs');
// ...
async function command(opts) {
  // ...
  let failed = false;
  const errors_dir = path.join(process.cwd(), 'errors');
  if (fs.existsSync(errors_dir)) {
    fs.rmSync(errors_dir, { recursive: true, force: true });
  }

  let errors_summary = {
    total_count: 0,
    per_package: [],
  };
  let violations_combined = [];

  for (const { relativeDir, resultText } of resultsList) {
    const package_dir = path.join(errors_dir, relativeDir);
    fs.mkdirSync(package_dir, { recursive: true });

    const stripAnsi = new RegExp(/[\u001b\u009b][[()#;?]*(?:[0-9]{1,4}(?:;[0-9]{0,4})*)?[0-9A-ORZcf-nqry=><]/g);
    const resultTextFormatted = resultText.replace(stripAnsi, "").split("\n");

    fs.writeFileSync(`${package_dir}/results.txt`, `${relativeDir}\n${resultTextFormatted.join('\n')}\n`);

    const violations_count = parseInt(
      resultTextFormatted
        .find(line => line.match(/✘ [0-9]+ problem.*/))
        .replace(/\s*✘\s*/, "")
        .replace(/ problem.*/, "")
    , 10);

    const violations_index = resultTextFormatted.indexOf("Errors:") >= 0
      ? resultTextFormatted.indexOf("Errors:")
      : resultTextFormatted.indexOf("Warnings:");

    let package_violations = [];

    resultTextFormatted.slice(violations_index).forEach(line => {
      if (line.match(/http/)) {
        const error_details = {
          count: parseInt(line.match(/\d+/)[0], 10),
          rule: line.match(/http.*/)[0]
        }
        package_violations.push(error_details);
        violations_combined.push(error_details);
      }
    });

    errors_summary.per_package.push({
      package: relativeDir,
      violations_count,
      package_violations: package_violations.sort((a, b) => a.rule.localeCompare(b.rule)),
    });

    errors_summary.total_count = errors_summary.total_count + violations_count;
    // ...
  };

  if (failed) {
    const combined_violations_sorted = violations_combined
    .reduce((acc, item) => {
      const already_exists = acc.findIndex(acc_item => acc_item.rule === item.rule);
      if (already_exists > -1) {
        acc[already_exists].count = acc[already_exists].count + item.count
        return acc;
      } else {
        return [...acc, item];
      }
    }, []);

    fs.writeFileSync(errors_dir + "/summary.json", JSON.stringify({
      count: errors_summary.total_count,
      errors_combined: combined_violations_sorted.sort((a, b) => a.rule.localeCompare(b.rule)),
      per_package: errors_summary.per_package.sort((a, b) => a.package.localeCompare(b.rule)),
    }, null, 2));
  }
}

Note:

ESLint rules are categorized by warning and error, but we have found that the categorization is highly opinionated and do not necessarily align to its actual severity so for that reason we merged warnings with errors.

ESLint offers a wide variety of options for formatters. I have reviewed each formatter but ultimately the stylish (default) formatter met all our needs. Although the sylish output required a bit of formatting, it is the only formatter that provides a relative path of the package location, count of errors per package, and provides source URLs to the violated rules.

Viewing and Updating the Summary

With the patch code shown above, our new lint command will output a summary file that will look something like this:

{
  count: 3,
  errors_combined: [
    {
      rule: "https://eslint.org/docs/latest/rules/foo",
      count: 2,
    },
    {
      rule: "https://eslint.org/docs/latest/rules/bar",
      count: 1,
    }
  ],
  per_package: [
    {
      package: "/packages/a",
      violations_count: 1,
      package_violations: [
        {
          rule: "https://eslint.org/docs/latest/rules/foo",
          count: 1,
        },
      ]
    },
    ...
  ]
}

From the generated summary file, we can extract the following information:

Total count of lint errors of the entire project
Total count of lint errors per rule of the entire project
Total count of lint errors of each package
Total count of lint errors per rule of each package

In addition to the summary file, the modified lint command will output each package results to an organized file structure:

errors/
  |-- summary.json
  |-- packages/
    |-- foo/
      |-- results.txt
    |-- bar/
      |-- results.txt

Ideally from here you would divide and conquer to resolve all of your project's rule violations. However, if your team lacks the capacity to address these errors right away, the best you could do is try to prevent the problem from growing.

The best way to keep track of your lint errors would be to keep the errors directory up to date in all of your pull requests. You could tell your developers to run the modified lint command, but having to rely on people remembering to do things is rarely a good idea, so you're left with two options: you could have your CI run the patched lint command and push the updated summaries to your pull requests - which would require your developers to pull their own branch after each commit they push - or you could use husky to force your local dev environments to run lint for you.

Follow their Getting Started guide for installing Husky to your Backstage project and configure a pre-push hook to run lint:

#!/usr/bin/env sh
. "$(dirname -- "$0")/_/husky.sh"

yarn lint
if git status | grep "errors/"; then
  echo "You forgot to run 'yarn lint'. The lint command was ran for you just now, commit the generated errors directory and git push again"
  exit 1
fi

Note:

Instead of having husky run before a git push, you could configure it to run every time a developer creates a new commit. Depending on the size of your Backstage project, the lint command could take quite a while. In this guide we opted for the pre-push hook to provide a more pleasant developer experience.

Having the pre-push hook create the commit and subsequently push the updated set of commits in a single step, without requiring developers to initiate an additional git push, would be a convenient approach. But, unfortunatey, the pre-push hook's mechanism involves calculating the specific commits to push at the moment the hook is activated. As a result, it's incapable of generating a fresh commit and incorporating that newly created commit within the confines of the hook itself.

Using the Data

Now that we have a better overview of our linting errors (and a way to keep it reliably up to date), we need to decide on a method of using that data.

The simplest approach to check if someone is introducing more lint errors would be to quickly check the git diff of their pull request for the total count in the summary file:

{
-  count: 3,
+  count: 5,
  errors_combined: [...],
  per_package: [...]
}

However, should a developer address a minor rule violation while simultaneously introducing a more severe error, the overall count of lint errors would remain unchanged. To avoid this scenario, you'd need to compare the cumulative count for each specific rule:

{
  count: 3,
  errors_combined: [
    {
      rule: "https://eslint.org/docs/latest/rules/foo",
-      count: 2,
+      count: 1,
    },
    {
      rule: "https://eslint.org/docs/latest/rules/bar",
-      count: 1,
+      count: 2,
    }
  ],
  per_package: [...],
}

And if you want to ensure that a particular rule isn't being resolved in one package while a new one is being introduced in another, you would have to take a look at the differences under the per_package property and review the git diff of the lint reports of each package.

Whether you choose to resolve your lint issues rule-by-rule or package-by-package, providing your developers with the capability to access the lint reports for each package in a file format offers significant convenience. This will enable them to search through the entire directy for particular rules and be able to view the details of the lint report without having to run the command.

Conclusion

If your project has accumulated a long list of lint errors, it's crucial to strategize an approach for addressing them. Your team might be able to allocate resources to fix those linting errors right away or they may need to chip away at it incrementally. Regardless of the approach, it would be beneficial for your team to be able to track the progress of their efforts - it's also very satisfying to see the total count number within the summary file decrease in your pull requests.

Upon successfully reducing your lint errors count to zero, you can promptly remove the patch and reconfigure your CI system to mandate linting as an obligatory check for all pull requests.

Deno is the best tool for maintaining NPM packages

Charles Lowell — Mon, 01 May 2023 05:00:00 GMT

TL;DR Even if you primarily run your JavaScript in Node, Deno is a fantastic tool for building and distributing packages on NPM (and everywhere else). This article shows how to publish modules to both https://deno.land/x and https://npmjs.org using nothing more than the deno CLI.

myth: You need to use node and npm , yarn , pnpm , et al. to publish packages to https://npmjs.org

Fact: Deno is perfectly capable of maintaining NPM packages. Studies conclusively show that maintaining NPM packages with Deno instead of Node is up to 900% more stress free.¹

As you may or may not know, here at Frontside we’re migrating toward Deno as our primary tool for JavaScript development. That means that we’ll not only use Deno from the get-go to develop our new JavaScript projects and publish them to deno.land, but over time we’ll also migrate our existing repositories that use Node as the primary development tool. But that doesn’t mean we’re rejecting the Node ecosystem – far from it!

We have a ton of NPM packages already developed that we and our clients critically rely on. Moving forward we want all of our new packages to be available to all JavaScript developers, not just those that happen to run their programs on Deno. And finally there is our work with Backstage, which comprises the lion’s share of our business. It runs on Node and will do so for years to come. In a nutshell: we have been and are going to continue to be deeply integrated with the Node community for the long haul.

But given this, how will we swap over to use Deno without losing what we have in terms of our existing install base of NPM packages and Node applications? Simple intuition tells us we should use Node to develop Node packages and Deno to develop Deno modules:

But as it turns out, Deno has excellent capabilities to build NPM packages that we use every day, allowing us to both have our cake and eat it too. With a slight tweak to how we think about versioning, we can get all the DX goodness of Deno, but our code is accessible to the widest possible audience of developers on both deno.land and NPM.

The key change in thinking that enables this transition is to move away from package.json as the final source of truth regarding the package version and to replace it instead with a humble git tag. Each released version will have a single tag to represent it, with the result that package.json goes from being authoritative to just another build artifact. In short, moving forward we’re following this mantra: If it doesn’t have a tag, it ain’t a release.

Tag once. Publish twice.

With that mental shift in place, we can implement the following scheme:

Wait until a release tag is pushed
Then use it to build and publish to both deno.land/x and npmjs.com

We’ll spend the rest of this blog post showing how we will do this for both platforms.

Stage 1: The release tag

For all projects, a release tag is a combination of a “version prefix” followed by a semantic version number. The most common version prefix is the letter ‘"v"’, so for example, the semantic version 1.3.5 would be tagged with "v1.3.5" . The beta version 1.6.0-beta.3 would be tagged with "v1.6.0-beta.3".

💡 While "v" is the most common version prefix for a single module, monorepos that contain multiple distributed modules will need to use a more specific version prefix for tags like “package-a-v,” which will result in release tags like package-a-v2.0.0

Stage 2: Build and Publish

2.1: Publish to deno.land

The deno.land build/publish is almost a gimme since it uses git tags natively. All we have to do is create and register the webhook for our module and we’re done! Moving forward, every time deno.land sees a release tag, it will snapshot our source at that tag and make that version available to users – in perpetuity.

2.2: Publish to NPM

There’s a little more work to be done getting our package onto NPM, but with a helping hand from Deno, we can do it in just two steps:

Write a script to take a version number and build an NPM package corresponding to it.
Implement a GitHub workflow to listen for a release tag and then invoke the script from (1) to build the package and publish it to https://npmjs.org

You might be asking yourself, “How are we going to take a bare ESM module implemented in TypeScript and just whip up a script that does all the things needed to end up with a valid NPM package?”

The short answer is that the script practically writes itself! That's because Deno has our backs with a brilliant little tool called dnt or Deno to Node Transform. dnt is downright amazing, and its capabilities are way undersold. Whether we're creating a new package that we want to publish everywhere or we are just migrating repositories from Node to Deno, dnt is the secret sauce we use that does all the heavy lifting for us.

dnt works by taking a single ES module entry point as the input and generating a feature-complete NPM package containing all its exports as the output. Along the way, it does all the terrible, tedious, un-fun, and error-prone things you hate about NPM-ing, such as maintaining the list of dependencies, compiling typescript, generating source maps and typings files, creating both commonjs as well as esm builds, and optionally vendoring any dependencies. The results work for users consuming the package from TS, JS, on the browser or the node – including every possible combination of those – and you didn't have to think about any of it. Magic!

I won’t go into all of the details because the dnt setup instructions are very thorough, but as an example, look at our build script at tasks/build-npm.ts in the GraphGen repository. It can build any version of our package, but here’s how we would invoke it to build version 1.0.0

$ deno run -A tasks/build-npm.ts 1.0.0
[dnt] Transforming...
[dnt] Running npm install...
added 6 packages, and audited 7 packages in 1s
found 0 vulnerabilities
[dnt] Building project...
[dnt] Emitting declaration files...
[dnt] Emitting ESM package...
[dnt] Emitting script package...
[dnt] Complete!

Now that we have the script to build an NPM package from source, the only remaining step is to add a GitHub workflow to be able to call it. Here is an example from the GraphGen repository. Each step below is linked to the relevant section in it, but the overall sequence is:

It’s about as straightforward as a GitHub workflow can get, and with it in place, we can now publish to two locations – just by creating a tag. (Trust me, it feels even easier when you try it yourself.)

Conclusion

Like many JavaScript developers, we have a heavy investment in the Node ecosystem, which means that we rely on NPM to distribute most of our code. However, because of Deno’s excellent support for NPM and NPM distribution, we’ve been able to forge ahead with migrating more and more of our repositories so they can use Deno as their primary JavaScript development tool – without sacrificing any of our NPM audience. Not only is Deno perfectly capable of managing NPM packages, but from our point of view it is actually a more pleasant experience than doing so with Node itself. I hope I’ve shown that if you’re thinking of giving Deno a spin it isn’t an either/or proposition. You can happily develop with Deno yet still publish your code to NPM, where it can be consumed just the way it always was by applications running on Node. That means that you can adopt it incrementally at a pace that makes sense for you and your organization.

So what are you waiting for?

Results obtained by conducting a survey of me.↩

Seven ways to build a single glass pane with Backstage

Taras Mankovski — Sun, 22 Jan 2023 05:00:00 GMT

The concept of a single glass pane is popular in the Cloud Native ecosystem because it promises to prevent fragmentation within many platforms where users have to interact with different systems to get their work done. Instead of going to one interface to see their builds, another to see their deployments, and yet another to see their logs, a single glass pane would make all of this information available in one place. Increasingly, organizations building platforms see Backstage as the one place to surface everything developers need to be productive.

Backstage is well positioned to be the single glass pane because it offers a user-friendly architecture designed with this in mind and a suite of plugins that make it easier to integrate with external systems. This blog post will review the Backstage architecture and cover seven separate ways to integrate with external systems to form a single glass pane in Backstage:

Explore Backstage Plugin
Entity Links
Custom React Component
Proxy Backstage Plugin
Custom Backend Plugin as a Gateway
Custom Processor
Custom Entity Provider

Each approach is valid and has both benefits and tradeoffs. Picking the most suitable approach depends on several factors related to security and integration complexity. This list is organized in order of increasing complexity, but as the complexity increases, so does the value you get from the integration. For first-time users, we recommend starting with the least complex integration as it will take the least time and effort while still delivering value. Based on the feedback you get from users, over time you can increase your integration complexity to generate more value.

Background

Client-Server Architecture in Backstage

Backstage has a fairly traditional client-side architecture with a sprinkle of a framework to make integrating easier.

A user interacts with the Backstage App via their browser. The Backstage App is a React-based Single Page Application. It consists of Backstage Plugins, each of which is a React Component. Each React Component can retrieve data by calling the Backstage Backend or calling external services directly.

The Backstage Backend is a Node.js service that uses Express.js to mount Backstage Backend plugins on the /api endpoint. There are several core plugins (like Catalog and Proxy plugins) that can be configured via TypeScript or YAML. The Backend plugins call external services via a variety of different protocols. The most common is HTTP, but there is no limit to the services that Backstage Backend plugins can connect to.

Option 1: Explore Backstage Plugin

Link to external services using the Explore Backstage Plugin

One of the most important goals of a single glass pane approach is to give users a unified view of the internal developer platform. This is achieved by using Backstage as a launch pad into the platform by making links to all external services available in Backstage. In effect, this means gathering and listing links to external services in the Backstage App. This is so simple and obvious that it’s easy to overlook how valuable this is for users. One of our clients had a user who maintained a list of 150 links that they shared with others. Without these links it was challenging to do their work. If you’re considering creating a single glass pane, your organization likely is experiencing a similar challenge.

The easiest way to add a list of links to external services is to use the Explore Plugin. It allows you to show a list of cards for each service with a link and organize these links into tabs. Instead of maintaining a list of links in say a Google Doc, developers will conveniently find these links in the Backstage portal.

Option 2: Entity Links

Eliminate unecessary steps by deep linking to external services using Entity Links

The Explore Backstage Plugin is helpful when linking to a webpage on an external service, but usually those external services have information related to a specific component in your catalog. For example, if your platform offers Kibana for viewing logs, you can add a link from Explore Backstage Plugin to Kibana’s home page. However, the user will still need to navigate Kibana to find the logs of a specific component. Navigating an external service from its home page is very time-consuming and difficult for users unfamiliar with these systems.

Show links in Entity Links Card by editing Entity Links metadata

A single glass pane can make it easier for users to navigate to specific pages in external services by providing URLs in the links property of the entity’s metadata. Backstage Catalog will show these links in the Links card on the Entity Page for that component. This technique provides native security protection by allowing the external system to ensure that the user is authenticated when the user arrives at the external system.

Option 3: Custom React Component

Give users a preview of content in external system by displaying it using a custom React component

An Entity Links Card is helpful when you want to show a simple list of links, but the user has to leave Backstage to see the content behind those links. This can be inefficient because it requires context switching to even simply glance at the information behind a link. We can provide a better user experience by embedding the information from the external service in the catalog’s Entity Page.

If we use Kibana as an example, instead of deep linking to a page on Kibana, we could fetch the last 100 log messages and display them directly on the Entity Page of a service component. A custom React Component would be responsible for making an API call to Kibana’s API and showing the returned information in the browser.

Option 4: Proxy Backstage Plugin

Use Proxy Backend Plugin to authenticate requests using a secret

Making API requests from a Custom React Component directly an external service is convenient when the user can authenticate directly against the external service and retrieve a token that can be used securely in the browser. Most services do not provide in-browser authentication. In this case, you can use Backstage’s Proxy plugin to call the external service on behalf of the user using a token stored in the Backstage Backend configuration.

Several Backstage Plugins integrate with external services this way (if you’re looking for inspiration on implementing this kind of integration, check out the CircleCI Backstage Plugin). However, be aware that once you configured the proxy plugin using a Backstage specific token, that token can be used by anyone who can call /api/proxy/<your_service> endpoint. Take care not to leak any information that should not be available to everyone authenticated with Backstage.

Custom Backend Plugin as a Gateway

Combine data from multiple services using a custom Backend Plugin as a gateway

There are various reasons why using the Backstage Proxy Plugin method to expose information to the Backstage App might not be enough. If you want to restrict the information users can retrieve from an external service, format the retrieved information, or combine information from multiple sources, a simple proxy is not enough. In these cases you need a custom Backstage plugin that is a gateway to your external service.

This gateway will respond to requests coming from the front end. It can inspect the requests to see if the user has permission to ask for the information they are requesting, make the necessary calls, format the retrieved response, and return everything as a single response.

Option 6: Custom Processor

Enrich entity information using Custom Processors

The five options we described so far all require a browser to access information in the external system. Some external systems (like LDAP) cannot be accessed from the browser. Integrating these kinds of systems while building a single glass pane with Backstage requires a different approach. Since accessing this information is not possible, we can instead use the Catalog’s REST API as a temporary cache. This cache is populated using Backstage Catalog’s processing loop.

You can configure the processing loop using Custom Processors. Custom Processors run on every entity stored in the catalog and can connect to external services to retrieve additional information. The retrieved information can then be used to enrich the entity. However, you must take care to ensure that you do not perform unnecessary asynchronous operations in your Custom Processors because it can eventually slow down the processing speed of other entities.

Option 7: Custom Entity Provider

Create new entities using a Custom Entity Provider

A Custom Processor can enrich existing entities, but where do entities come from in the first place? Entities come from Entity Providers. An Entity Provider is an extension point of the Catalog that allows you to commit new entities to the processing loop. Entity Providers extract information from external services, format them as entities and commit them for processing. Entities committed by Entity Providers will execute each Custom Processor relevant to that entity kind.

Entity Providers follow two ingestion strategies: direct or indirect. Indirect Entity Providers discover URLs of possible sources of entity information and commit discovered URLs as an entity of Location kind. Indirect Entity Providers in Backstage usually discover catalog-info.yaml files and expect Custom Processors to actually read these files and extract entities from them.

Conclusion

In this blog post, we covered an overview of the Backstage architecture and seven separate ways to integrate services into Backstage to create a single glass pane. In future blog posts, we’ll expand on techniques you can use to create a single glass pane with a unified data graph of all developer assets.

If you would like Frontside to help your team put these techniques into practice, check out our Backstage Enterprise Support and consulting services.

Dynamic Github Action Jobs

Jacob Bolda — Mon, 12 Dec 2022 05:00:00 GMT

Dynamic Github Action Jobs

Github Actions is a CI/CD platform integrated directly with your github.com repository. It is a popular choice to build and test your code within a pull request or on a commit to a branch. It also has many other options for triggering a workflow.

As a team embraces their Github Action implementation the CI+CD run times can explode. This leads a team towards tuning their implementation and only running the specific jobs that are required. Naturally in this evolution, the need to build workflow context dynamically exponentially increases. Keeping all of the context in a workflow file becomes quite difficult to maintain, and is limited by static values.

The workflows are written in YAML. The event(s) that triggers the workflow are specified with the on parameter. Each workflow file has one or many jobs with a named parameter, in this example run-linters. Each job has a list of steps which is the actions or commands that are run. This job has three steps which install the dependencies and run the lint script.

name: Lint

on: push

jobs:
  run-linters:
    name: Lint
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2
      - uses: volta-cli/action@v1

      - name: Install Node.js dependencies
        run: npm ci

      - name: Run linters
        run: npm run lint

Within a GitHub action workflow, steps can pass data to the following steps. A step can set output, and the following steps can use that it as a variable. In the example below, the named step, Get Version, sets an output value. The :: syntax is picked up by the Github Action runner when output to stdout, ie. the terminal. More information about this syntax can be found at the workflow commands Github docs. The id is also set in this step as vars. This is important as this is the variable the following steps use to retrieve the value. The step named Build immediately following sets an env using steps.vars to refer to the Get Version step.

name: Release to NPM
on: [push]

jobs:
  release:
    runs-on: ubuntu-latest

    steps:
      - name: checkout
        uses: actions/checkout@v3
      - name: setup deno
        # uses: denoland/setup-deno@v1
        uses: denoland/setup-deno@004814556e37c54a2f6e31384c9e18e983317366
        with:
          deno-version: v1.x
      - name: Get Version
        id: vars
        run: echo ::set-output name=version::$(echo ${{github.ref_name}} | sed 's/^v//')
      - name: Build
        run: deno task build:npm $NPM_VERSION
        env:
          NPM_VERSION: ${{steps.vars.outputs.version}}
      - name: Publish
        run: npm publish --access=public
        working-directory: ./build/npm
        env:
          NODE_AUTH_TOKEN: ${{secrets.NPM_TOKEN}}

This concept also applies to jobs. A step can set an output, and the job can take that step output and set it as a job output. This allows you to do actions dependent on previous job output. "How might this be useful though?", you ask.

One way to level up your jobs is a parameter called strategy. It allows you to pass an argument, matrix, to run multiple jobs in parallel. This takes an array or multiple arrays of arguments which will then be combined into a number of jobs. In the example below, we specify a matrix with two sets of values. The platform is referenced in the runs-on and node-version is used by the volta-cli/action. This specific array will produce a job set of consisting of six jobs run in parallel, e.g. ubuntu-latest + nodejs@16.

name: Test
on: [pull_request]

jobs:
  test:
    runs-on: ${{ matrix.platform }}
    name: ${{ matrix.platform }} test node@${{ matrix.node-version }}
    strategy:
      fail-fast: false
      matrix:
        platform: [ubuntu-latest, windows-latest]
        node-version: ['14', '16', '18']
    steps:
      - uses: actions-rs/toolchain@v1
        with:
          toolchain: 1.63
      - uses: actions/checkout@v2
      - run: yarn install
      - run: yarn build
      # specifically running this after build
      # using the n-api, the default node on github actions
      # should build a .node that works in any of the node
      # versions that we have specified in the matrix
      - uses: volta-cli/action@v1
        with:
          node-version: ${{ matrix.node-version }}
          yarn-version: 1.22.19
      - run: yarn test

As an example in a monorepo, you may have ten different packages. You can give it an array which includes a list of each of the packages and an array of nodejs versions. The jobs that would be then created based on these two matrix inputs and would produce 20 jobs, 10 services by the two nodejs versions. It enables parallel processing and may increase the speed of your CI runs. This can also increase the transparency as you see the separate jobs in the status section of a pull request.

![[dynamic-github-action-jobs-status-checks.png]]

The downside of the static array of values is having multiple sources of truth. If you have multiple workflows, this matrix needs to be replicated across each and kept in sync. The matrix accepts an argument called include which allows you to specify additional jobs over and above the jobs derived from the matrix arrays or additional metadata each job(s).

The Github Actions documentation has an advanced example. This following example would add a matrix.color to each job with a value of green. There would be four total jobs each with a { fruit, animal, color }.

strategy:
  matrix:
    fruit: [apple, pear]
    animal: [cat, dog]
    include:
      - color: green

A strategy is able to specify only an include. The following example create two jobs, each with { color, fruit }. The first job would be { color: "green", fruit: "apple" } and the second job would be { color: "blue", fruit: "pear" }.

strategy:
  matrix:
    include:
      - color: green
        fruit: apple
      - color: blue
        fruit: pear

This is where a few built-in functions within GitHub Actions can become immensely useful using output passed between jobs and a function to convert that output into a matrix value. We can use this to our advantage; compute the array of jobs ourselves and pass it to this include argument.

Through the toJSON function, the matrix argument can receive a stringified object. The toJSON function takes a string and outputs a value that the job is able to use. This allows us to programmatically, in a previous job, determine the matrix of jobs to run.

It is particularly powerful in monorepos where you can use static analysis to determine which code paths have changed, and only test those. With the monorepo helper Nx, for example, we can run a function which describes the code that has been affected by a change in pull request, nx affected. Using this output, we can parse it into an object and pass it into our matrix.

The object can be created however it is most convenient, as long as it can be output to the terminal. For an example using nodejs, we would log out the output of a JSON.stringify(). The script shown below, when run, would create three jobs with a matrix.package value being passed into each.

const matrixList = ['package-a', 'package-b', 'package-c'].map((pkg) => ({
  package: pkg,
}));

const includeStatement = { include: matrixList };
console.log(`::set-output name=matrix::${JSON.stringify(includeStatement)}`);

We use this script in the following workflow. It has two jobs: generate-matrix and test. In the generate-matrix job, we run the matrix-script.js shown above which logs the string to stdout. The ::set-output name=matrix:: is a special function within Github Actions which tells the runner to set this as an output value for this step. We assign it to a variable by setting the id: set-matrix. Finally, we can set it as an output for the whole job by specifying it in outputs.

The following job can use the output from the previous job by setting the job ID in the needs property. The test job uses needs: [generate-matrix] and can use the values via the needs elsewhere in the job. It has a conditional which determines if the job should run or skip with if: needs.generate-matrix.outputs.matrix != ''. It is referenced with the needs.<job id>.outputs.<property from job id outputs>.

jobs:
  generate-matrix:
    name: Generate Job Matrix
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    steps:
      - uses: actions/checkout@v3
      - run: yarn
      - run: node ./matrix-script.js
        id: set-matrix

  test:
    name: Check ${{ matrix.nickname }}
    runs-on: ubuntu-latest
    if: needs.generate-matrix.outputs.matrix != ''
    needs: [generate-matrix]
    strategy:
      fail-fast: false
      max-parallel: 6
      matrix: ${{fromJSON(needs.generate-matrix.outputs.matrix)}}

    steps:
      - uses: actions/checkout@v3
      - run: yarn
      - run: yarn workspace ${{ matrix.package }} test

Building on this functionality, we can level up our script, matrix-script.js with static code analysis. As we mentioned previously, Nx has a command called nx affected. We can wire this into our script similar to the following. This would return a list of affected packages. (Note this is a generator function expected to run within an effection context, so you may not be able to copy and paste this directly.) This would give you a dynamic array that can be used in the script rather than directly hardcoding the array as ['package-a', 'package-b', 'package-c'].

export function* affectedList() {
  const nxBase = process.env.NX_BASE ?? 'main';
  const nxHead = process.env.NX_HEAD ?? 'HEAD';

  // if you want to check it locally with uncommitted changes
  // const command = `nx affected:libs --plain --uncommitted`;
  const command = `nx affected:libs --plain --base=${nxBase} --head=${nxHead}`;
  const { stdout, stderr, code } = yield exec(command, {
    cwd: '..',
    env: process.env,
    shell: true,
  }).join();

  console.error(stderr);
  if (code !== 0) throw new Error(`exited with code ${code}`);
  const affected = stdout
    .split(' ')
    .map((x) => x.trim())
    .filter((x) => x.length > 0)
    .join(',');

  return affected;
}

With the additional context required to fuel nx affected, the following workflow would create the extra context via nwql/nx-set-shas, derive the nx affected, pipe the generated matrix into test job, create a job for each affected package and run the test for the package. The nx command yarn nx run ${{ matrix.package}}:test will have the ${{ matrix.package }} populated with the value passed into each job. The command run for the job with { package: 'package-a' } would in turn be yarn nx run package-a:test.

jobs:
  generate-matrix:
    name: Generate Job Matrix
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
      nxBase: ${{ steps.nx-sha.outputs.base }}
      nxHead: ${{ steps.nx-sha.outputs.head }}
    steps:
      - uses: actions/checkout@v3
        with:
          # We need to fetch all branches and commits so that Nx affected has a base to compare against.
          fetch-depth: 0
      - name: Derive appropriate SHAs for base and head for `nx affected` commands
        id: nx-sha
        uses: nrwl/nx-set-shas@v2
      - run: yarn
      - run: node ./matrix-script.js
        id: set-matrix

  test:
    name: Check ${{ matrix.nickname }}
    runs-on: ubuntu-latest
    if: needs.generate-matrix.outputs.matrix != ''
    needs: [generate-matrix]
    strategy:
      fail-fast: false
      max-parallel: 6
      matrix: ${{fromJSON(needs.generate-matrix.outputs.matrix)}}

    steps:
      - uses: actions/checkout@v2
      - uses: volta-cli/action@v1
      - run: yarn
      - run: yarn nx run ${{ matrix.package }}:test

With this workflow and creating our own array of jobs, it enables tight control of what and when CI runs. This can save on time and build minutes, as well as increase transparency through the parallel CI runs. With a set of workflows that builds, tests, creates docker files and helm charts and finally deploys, tuning the CI+CD is required to merge more than a single pull request or two a day.

We can use static code analysis to drive a CI setup that only builds and tests the code paths that have changed. It provides the most feedback in the shortest amount of time. From this, we can enable CD to deploy only the services that have changed into a QA environment and save this metadata for production deploys.

Want to learn more? We have a great community in Discord; come and say hello! We like to help and discuss all kinds of tech and code topics.

Ingestion tests in Backstage Part 2: Safely refactoring an LDAP integration

Charles Lowell — Mon, 05 Dec 2022 05:00:00 GMT

Heads up: This is the second part in a series on how to achieve real confidence in your Backstage Ingestion via testing. We’ll be relying on the techniques introduced in the first blog post to test a non-trivial external application. What follows may not make much sense without a grasp of those techniques, so I highly recommend starting with Part 1 first and then coming back.

In our article on testing your Backstage catalog ingestion, we discussed the technique of using the concepts of structured concurrency and eventual consistency to test a backstage server solely via its external interfaces: as an operating system process and an http server. The payoff for organizing your tests that way is that they end up at the ideal sweet spot intersection: replicating how the server actually runs and behaves in production while still maintaining desirable test properties such as speed, isolation, and repeatability.

Rather than reaching into the guts of your ingestion code and testing the inputs and outputs of individual viscera, this approach instead embeds your server as a single holistic unit directly into your test cases. This means that not only can you test anything in your server using the exact same test harness, but also you are free to re-organize the internals of the server as you see fit, and your test cases need never change.

In this article, we’ll demonstrate why this is so important by refactoring in order to upgrade a Backstage server from a legacy LDAP org processor to a new LDAP entity provider – all without changing a single line of the test case.

If you recall, the fundamental strategy of the basic ingestion test case for locations defined in a fixed YAML was as straightforward as we dared to make it:

Start a Backstage server;
Let it run for a while; and
Check to make sure that the entities we expect were ingested.

We’ll use the exact same strategy as before with the only difference being that instead of checking that service components are ingested from our YAML files, we’ll check that our user entities are properly ingested from our LDAP server.

Testing Architecture featuring a simulated LDAP server

Using a simulator as a test double means that we get all the benefits of isolation and repeatability when using a mock or stub, but without having to sacrifice any of the confidence in the viability of our test because it’s actually using 100% of the production code with no additives or substitutes.

In fact, we’ve developed an LDAP simulator just for this purpose, which we can use inside of our test case. With this tool in hand, we can start up a simulated LDAP server instantaneously – right before we start our Backstage server – so the only thing that is different is that our test goes from this:

yield createBackstage();

to this:

yield createLDAPServer();
yield createBackstage();

That was easy! Now that we have our very own LDAP server embedded in our test case, all we have to do is seed it with user and group data, and then use our convergent assertions to confirm that those records were ingested in the catalog:

let catalog: CatalogApi;

beforeEach(function* () {
  yield createLDAPServer({
    users: [
      {
        cn: 'cowboyd',
        uid: 'cowboyd',
        name: 'Charles Lowell',
        uuid: 'xyz123-cowboyd',
        mail: 'cowboyd@example.com',
        password: 'password',
        avatar: 'https://avatars.dicebear.com/api/open-peeps/cowboyd.svg',
      },
    ],
  });
  catalog = yield createBackstage();
});

/// other catalog tests

it.eventually('ingests users from LDAP into the catalog', function* () {
  let user = yield catalog.getEntityByRef('user:cowboyd');
  expect(user).toMatchObject({
    spec: {
      profile: {
        email: 'cowboyd@example.com',
        picture: 'https://avatars.dicebear.com/api/open-peeps/cowboyd.svg',
        displayName: 'Charles Lowell',
      },
    },
  });
});

Upgrade with confidence

One of the main points I’ve been stressing in these articles is that this style of test only ever uses the public API of Backstage, which means that no matter what is going on under the covers, our test case verifies the most important aspect of its behavior: what it will do when you actually use it. And because our test only uses the “outside” of Backstage, it means that we have a free hand to change whatever is inside without worrying that we might break our tests.

In the example repository, our initial LDAP integration used a processor combined with a custom user transformer in order to map the fields from our LDAP user entry into a Backstage user entity. Here is our transformer that maps the name and avatar fields from our entry into the spec.profile.displayName and spec.profile.picture fields of our user entity:

async userTransformer(vendor, config, entry) {
  let user = await defaultUserTransformer(vendor, config, entry);
  return merge(user, {
    spec: {
      profile: {
        displayName: vendor.decodeStringAttribute(entry, 'name')[0],
        picture: vendor.decodeStringAttribute(entry, 'avatar')[0],
      }
    }
  });
}

But this is the old way. The new way is to use an entity provider, and a simple set of field mappings in your app-config.yaml. We want the transformation to look like this:

users:
  map:
    displayName: name
    picture: avatar

Let’s start with a passing test suite:

PASS  src/catalog.test.ts (18.847 s)
  catalog ingestion
    ✓ can connect to the catalog (5348 ms)
    ✓ ingests the artist lookup component  (4469 ms)
    ✓ ingests the example user from ldap into the catalog (4298 ms)

Test Suites: 1 passed, 1 total
Tests:       3 passed, 3 total
Snapshots:   0 total
Time:        19.208 s
Ran all test suites.

If we first migrate our catalog to use the entity provider instead of the processor, but don’t add the field mappings yet, our test suite automatically fails:

FAIL  src/catalog.test.ts (21.039 s)
  catalog ingestion
    ✓ can connect to the catalog (4318 ms)
    ✓ ingests the artist lookup component  (4700 ms)
    ✕ ingests the example user from ldap into the catalog (9063 ms)

  ● catalog ingestion › ingests the example user from ldap into the catalog

    expect(received).toMatchObject(expected)

    - Expected  - 2
    + Received  + 1

      Object {
        "spec": Object {
          "profile": Object {
    -       "displayName": "Charles Lowell",
    +       "displayName": "cowboyd",
            "email": "cowboyd@example.com",
    -       "picture": "https://avatars.dicebear.com/api/open-peeps/cowboyd.svg",
          },
        },
      }

      45 |   it.eventually('ingests the example user from ldap into the catalog', function*() {
      46 |     let user = yield catalog.getEntityByRef('user:cowboyd')
    > 47 |     expect(user).toMatchObject({
         |                  ^
      48 |       spec: {
      49 |         profile: {
      50 |           email: 'cowboyd@example.com',

      at Object.<anonymous> (catalog.test.ts:47:18)
      at produce (../../../node_modules/@effection/core/src/future.ts:115:15)
      at Promise.race.then.produce.state (../../../node_modules/@effection/core/src/controller/promise-controller.ts:11:9)
          at runMicrotasks (<anonymous>)

Test Suites: 1 failed, 1 total
Tests:       1 failed, 2 passed, 3 total
Snapshots:   0 total
Time:        21.514 s
Ran all test suites.

Not only has it detected that something is amiss, but it has also told us exactly what went wrong. The record has been found, but the spec.profile.displayName and spec.profile.picture fields are no longer being populated correctly. However, if we add those mappings to our app-config.yaml the tests re-run and pass again:

PASS  src/catalog.test.ts (17.195 s)
  catalog ingestion
    ✓ can connect to the catalog (4962 ms)
    ✓ ingests the artist lookup component  (5102 ms)
    ✓ ingests the example user from ldap into the catalog (4223 ms)

Test Suites: 1 passed, 1 total
Tests:       3 passed, 3 total
Snapshots:   0 total
Time:        17.645 s, estimated 22 s
Ran all test suites.

We completely changed the mechanism for ingestion of users from LDAP, and yet our tests did not change by a single line. Don’t believe me? Check the diff!

Focus on what matters

We know tests are important, but they can be so difficult to set up and such a nightmare to evolve along with your codebase that most of the time we don’t even bother. But a workable and efficient solution is to imagine the simplest, most durable test you can, and then put the work in to attack the complexity – separating that from being able to express that simple, durable test in code. In the case of ingestion, the simplest thing that could possibly work is to turn it all on and see what happens, and take a “wait-and-see” attitude toward complexity.

Deno is "blazing fast" for humans

Charles Lowell — Thu, 01 Dec 2022 05:00:00 GMT

Are you old enough to remember back in the day when Node.js was the new kid on the block? I am. I remember the excitement of having a dead-simple way to run JavaScript on the server. I remember being intrigued by the prospect of running the same code run on both the frontend and the backend. But above all else, I remember the promise that it was going to be fast. Like, really, really fast. And it was…. sort of.

The code ran quickly enough. The separation of every single I/O operation via callbacks meant absolutely no wait time was introduced into a program without you explicitly asking for it, and compared with other prominent scripting languages of the day like Ruby and Python it was quite the coup. I still recall the first time I ran a node-based CLI and being impressed by the complete absence of any gap between when the command was invoked and when it produced its first output. Here was a tool which not only felt familiar to JavaScript developers, but which also let you build programs that felt really responsive. The response time was so far below the threshold of perception that it soon became customary (to the point of parody) to label every new application or library written for Node.js as “blazing fast.”

But a funny thing happened over time as the ecosystem of “blazing fast” applications ballooned in size: while they ran faster than ever, individually they each became slower and more painful to write and maintain. I often wonder what my 2010 self would say to me if he learned about how regularly I would spend 30 minutes of my day just waiting for a JavaScript codebase to build – sometimes multiple times a day! What would his face convey as I explained to him all of the daily “necessities” that gradually and relentlessly chipped away at his joy?

“yarn install," I imagine myself saying to him a tad defensively. “It’s absolutely critical to maintain dependencies. You can’t live without it.”

I see him staring back at me impassively, saying nothing.

“TypeScript,” I offer. “JSX too. You don’t know about either of these yet, but trust me, they are incredible time-savers.”

Silence. I’m beginning to feel uneasy.

“Babel?” I say almost sheepishly. “Webpack?”

His face is impassive, save for the single tear that slowly slides down the side of one cheek.

“ESLint!” I shout. “Prettier!” I’m aware of the shrill edge creeping into my voice, but I can’t help it. I’m beginning to feel desperate.

“Thirty minutes?” he whispers, his voice nearly inaudible.

“Relay Compiler!” I plead. “Look, I know that Jest has its own completely parallel build toolchain that is unrelated to production, but it’s a really handy, popular tool!”

He glances down at his watch, shakes his head, and turns away from me.

“Why can’t he see it?” I think to myself. Of course, each little tool we use in our projects these days comes with its own small overhead, and yes, they come with their own configuration mechanisms, and no, they don’t always play nice with each other and cause you to lose a day here and there untangling them from each other, but surely he can see the value in them can’t he?

Unfortunately, the dispute between us can never really be resolved to either of our satisfaction and the reason why is that we’re both right.

From my perspective in 2022, I can plainly see all the clear wins that things like TypeScript and JSX offer, not to mention modern comforts like an established testing framework that Jest provides. But just as relevant to that class of time savers is the feeling of fleetness that Node imbued. And that's his point: how many hours – no, days – have I spent waiting for the code to build? I’ll never get those back.

Luckily now there’s a tool that both of us can get behind. Deno brings together the best parts of 2010 and 2022 and fuses them together for an outstanding developer experience without compromises. You have all the power and speed of a modern JavaScript runtime but none of the time-intensive install and build steps that you have to intermittently run during every portion of the development process. With Deno, you just get the code and run it.

Why and how is that possible? Because Deno is more than a JavaScript runtime. It’s a fully formed software development kit that contains all the tools you need baked right in.

Want to just run a JavaScript file? Easy.

Or perhaps TypeScript? Same.

Linter? Formatter? Yup.

Bundler? Task Runner? Test Framework? Language Server? Check. Check. Check…. and let’s see… yup, check.

It even comes pre-rolled with a cross-compiler to turn scripts into platform binaries, and there’s more pieces of the SDK puzzle being added to it every day.

As a result of all the wins baked in, working on a Deno codebase feels nearly frictionless. The time between when you clone a repository to when you’re running tests or even starting it up is barely noticeable.

So yeah, it’s time to resurrect the cliché – only this time it’s really true. Deno is blazing fast. It runs code quickly to be sure, but even more important than the rate at which it can shove bytes through a CPU is how Deno optimizes my time – and I appreciate that more than anything. That’s what I mean when I say that Deno is blazing fast and why it’s quickly become my tool of choice for JavaScript development.

What Is Spotify Backstage: a developer portal overview

Javier Lainfiesta — Tue, 17 May 2022 05:00:00 GMT

Backstage is a powerful open-source developer portal platform. What started as an internal tool for the Spotify developing team has become a versatile option for creating and managing developer portal ecosystems. Backstage stands out for its flexibility and high degree of customization. Below I want to share some features Backstage offers.

What can Backstage do for you and your team?

Let's do a short exercise. Imagine you have several teams distributed around the world working on different aspects of an application. Some are working on the mobile app, others on a web app, and many people are working on microservices. They all overlap in one way or another. The web app might need a GraphQL API provided by another team. The GraphQL API team might need to consume microservices. Teams maintaining a microservice might need to access data. Each of these teams works autonomously but they rely on the software created by others in the organization.

The autonomy allows developers to move fast, but it also brings unexpected problems. Over time individual services diverge and critical information gets lost. Developers wind up spending more time gathering information and finding resources than actually coding.

The team at Spotify calls this phenomenon the Speed Paradox. In simple terms, the faster you grow, the more fragmented your software ecosystem becomes. And that slows you down again.

That brings us to the main problem Backstage solves for many organizations: centralizing services and standardizing your tooling. It may sound counterintuitive, but uniformity breeds creativity. Having an integrated development environment from end to end gives your developers the freedom and resources they need to solve any complex problem fast.

Backstage helps teams by giving a uniform overview of all the services, libraries, websites, ML models, etcetera. This way, a team can manage all the software they own, including deployments, data pipelines, and libraries. The platform also makes it easier to locate info related to ownership and dependencies by other teams.

Technical speaking, what can we find in Backstage?

By now, the skeptic in you might be thinking: “All of the above sounds great, but how exactly does Backstage accomplish all of that?”

Through Backstage’s frontend application, team members can find not only the information they’re looking for but discover unknown resources and scaffolding for new projects. The core Backstage features include Software Catalog, Software Templates, Search, TechDocs, and Plugins. In that sense, the most fundamental use case of Backstage is information retrieval and accessing resources like documentation, ownership information, and status of running services. Let’s take a brief look at each feature.

Software Catalog

This feature helps your team find the metadata and ownership information of all the software they are working on. The catalog is based on metadata YAML files. It offers one place to view all the external tools to manage your code by tracking third-party software.

Software Templates

This powerful tool lets your organization create projects from a predefined reference template with forms that the team can fill out. The templates help standardize your tooling when the team starts coding a new project. Developers can use it for onboarding guides and good practices, and organizations can push their templates to GitHub or GitLab.

Search

The search bar in Backstage was designed to reduce the time teams spend looking for context and switching between sources of information. Backstage Search lets you customize the search tool for your specific needs with modular backend and frontend components. You can integrate your plugins, wiki, or even Stack Overflow, so all the results appear on the same screen.

TechDocs

Creating technical documentation is easy with this Backstage feature. TechDocs allow collocating documentation with the project source code to make it convenient for both developers working on the project and consumers of the project. This feature significantly reduces the time looking for documentation so it can be used.

Plugins

The best way to extend and extract all the value Backstage can offer your organization is with plugins. The community behind the Backstage open-source project is constantly working on plugins to benefit all the organizations that implement it. Plugins are React components and optional Node.js microservices that power these components. It’s also possible to build your own plugins.

Unleash the power of Backstage

Although these features are part of Backstage’s core offerings, the real power of this platform is in how customizable it is. You can create tailor-made developer portals and introduce all kinds of specific functionalities like testing, data analysis, etc. What you see here is just the tip of the iceberg.

If you are interested in learning more, check out Frontside’s blog or consider subscribing to our newsletter. As proud members of the Backstage development community, we are constantly working to extend this platform's power.

Want to learn more? We have a great community in Discord; come and say hello! We like to help and discuss all kinds of tech and code topics.

Tutorial: Using Github Webhooks with Backstage Entity Provider

Min Kim — Tue, 03 May 2022 05:00:00 GMT

Entity Providers are a more scalable and robust alternative to Entity Processors. The Backstage team introduced Entity Providers to solve problems that big deployments of Backstage were experiencing with the ingestion pipeline. If you take a look at their documentation on The Life of an Entity, it illustrates how your new ingestion pipelines should be structured.

Before the Entity Providers came into the picture, the preexisting processors essentially took on the roles of both the entity providers and the processors described in the documentation. I bring this up as it may provide clarity to anyone confused by some of the overlapping functionality of the current processors and entity providers in the Backstage plugins; some of those processors have not yet been updated to adapt to the new proposed ingestion pipeline.

Getting back to why Entity Providers were introduced, there are some common issues with ingestion processors:

The processing queue would get filled with no-op processing, putting pressure on external systems. In some cases, it leads to rate limiting requests.
Writing performant custom processors required implementing caching, which many teams did not.
Failure in external service leads to the creation of orphan entities, which disappear components from the catalog.

Entity providers eliminate these problems by giving developers complete control over the execution of ingestion:

Entity Providers do not have an implicit queue that drives their execution; instead, developers specify the mechanism that drives each entity provider. A driver for an entity provider can be a simple callback that runs on an interval, an event listener triggered by a Web Socket connection, or a response to the HTTP request.
There is no need to cache requests because there is no implicit orphaning of entities. Entities are only mutated when the Entity Provider calls commitMutation on the connection. The developer can control update frequency by configuring the mechanism that drives the entity provider.
Entity Providers automatically handle efficiently merging small changes to many entities without explicitly applying deltas. It’s possible to use just the deltas if the data source provides them.

Since entity providers do not have an implicit queue, you’ll need to specify what drives each entity provider. You could use a task scheduler and specify the frequency of entity provider executions. Ideally, your entity provider would respond to changes in the original data source and remain idle all other times. Developers can accomplish this with webhooks and streaming.

In this tutorial, we'll walk you through the steps of adding GitHubOrgEntityProvider to your catalog and then show you how you can configure Github Webhooks to trigger mutations to your Backstage database.

Adding an Entity Provider

Assuming you already have your own instance of Backstage, let's install the necessary packages for adding the Github Org Entity Provider:

yarn workspace backend add @backstage/plugin-catalog-backend-module-github @backstage/integration

Then you can import the provider to your catalog builder in packages/backend/src/plugins/catalog.ts. Be sure to replace the orgUrl with your own:

- import { CatalogBuilder } from '@backstage/plugin-catalog-backend';
+ import { CatalogBuilder, EntityProvider } from '@backstage/plugin-catalog-backend';
...
+ import { ScmIntegrations, DefaultGithubCredentialsProvider } from '@backstage/integration';
+ import { GitHubOrgEntityProvider } from '@backstage/plugin-catalog-backend-module-github';

export default async function createPlugin(
  env: PluginEnvironment,
): Promise<Router> {
  const builder = await CatalogBuilder.create(env);
  builder.addProcessor(new ScaffolderEntitiesProcessor());

+  const integrations = ScmIntegrations.fromConfig(env.config);
+
+  const githubCredentialsProvider = DefaultGithubCredentialsProvider.fromIntegrations(integrations);
+
+  const gitProvider = GitHubOrgEntityProvider.fromConfig(env.config, {
+    id: "github-org-entity-provider",
+    orgUrl: "https://github.com/my-organization", // 🚨 REPLACE
+    logger: env.logger,
+    githubCredentialsProvider
+  });
+
+  builder.addEntityProvider(gitProvider as EntityProvider);

  const { processingEngine, router } = await builder.build();
  await processingEngine.start();

  return router;
}

If you try to run your Backstage app, you should notice nothing has changed in your catalog. As mentioned earlier, developers must provide the mechanism that will drive the entity provider. For the sake of this example, let's add the GitHubOrgEntityProvider's read() at the end of our catalog builder:

  await processingEngine.start();
+  await gitProvider.read();

  return router;
}

When you restart your Backstage, you should see the members and teams of your organization. If you do not, you should check the permissions of your Github authentication. Whether you're using a personal access token or a Github App, you need to make sure it has permissions for read:user and read:org.

Calling read() towards the end of the catalog building process will only update your database once during deployment so now we're going to configure a webhook to trigger the updates.

Configure Github Webhook

Let's start by creating your webhook on Github. From your Github organization's settings page, click Webhooks in the side bar and then Add webhook.

github-webhook-section-screenshot

You can get a Payload URL from smee.io - this is a service that proxies payloads from your webhook for local development.

Set the content type to application/json and let's specify the events we want by selecting Let me select individual events. The GitHubOrgEntityProvider adds users and teams to your catalog so the webhook events we want to receive from Github are Orgnaization and Teams.

For your actual deployment, you'll want to modify the Payload URL from the smee URL to the URL of your live Backstage app.

Once the webhook is added, you should follow the instructions displayed on smee.io to install their client and use http://localhost:7007/api/catalog/github/webhook as the target URL.

Next, update your catalog builder so that it runs read() when a webhook event is posted to /github/webhook:

  await processingEngine.start();
-  await gitProvider.read();
+  router.post("/github/webhook", async (req, _res) => {
+    const event = req.headers["x-github-event"];
+    if (event == "membership" || event == "organization) {
+      await gitProvider.read();
+    }
+  })

  return router;
}

If you run the smee client and your Backstage app, you'll see that the entity provider will update the database only when a webhook event is posted by Github.

If you have concerns of the possibily of a webhook being missed, you might want to consider using the Backstage task scheduler to run read() once a day for that extra assurance.

What's next?

In this tutorial we quickly went over the steps of adding an entity provider to your catalog and using smee to proxy webhook events to your local environment, but this is just the beginning!

If you look at the implementation of GitHubOrgEntityProvider, the read() function queries data directly from Github and runs a full mutation. In its current state, depending on the size of your organization, your read() function might end up triggering way too frequently - resulting in too many Github requests and performing a full mutation of your database each time.

When you create your own custom entity provider, you will want to create a function that applies a delta mutation just from the data received from the webhook events. You can read more about the mutation types here.

Testing Backstage Catalog's ingestion

Charles Lowell — Thu, 24 Mar 2022 05:00:00 GMT

A test is a mechanism to answer a simple yes or no question. In my case, the question I want to ask is, "When my Backstage server runs, is it going to ingest stuff into the Catalog properly?"

It seems to me that there is only one way to find an honest answer:

Start a backstage server.
Let it run for a while.
Query its Catalog API over HTTP to see if it contains the records we want.

It's a small ask that can nevertheless be quite difficult to satisfy. Not only are operating system processes difficult enough to manage already, but also "let it run for a while" is an easy thing to say that is quite hard to put into practice.

Fortunately, by using structured concurrency coupled with the novel application of eventual consistency from database theory, you can write tests for your backend that are expressive of this question and are both fast and reliable. I'll show you the technique what we use at Frontside with large Backstage codebases to achieve this.

For this article, I've applied the same technique to a sample application scaffolded with npx @backstage/create-app (you can find it in github.com/cowboyd/backstage-testing-example).

Our approach works the same whether we're testing the humblest starter kit or the most heavily customized Backstage instance because the tests only interact with Backstage as an OS process and an HTTP server.

Here is what a test looks like to check that the sample data is ingested into the Catalog. I'll go over each section in turn, but as you read it, take note that the test really is telling you what it is doing.

import type { CatalogApi } from '@backstage/catalog-client';
import { describe, beforeEach, it } from '@effection/jest';
import { createBackstage } from './support';

describe("catalog ingestion", () => {
  let catalog: CatalogApi;

  beforeEach(function* () {
    catalog = yield createBackstage();
  });

  it.eventually('ingests the artist lookup component ', function*() {
    const component = yield catalog.getEntityByRef('component:artist-lookup');
    expect(component).toMatchObject({
      metadata: {
        name: 'artist-lookup',
        description: "Artist Lookup",
        tags: ["java", "data"],
      },
      spec: {
        type: 'service',
        owner: 'team-a',
        system: 'artist-engagement-portal'
      }
    });
  });
});

This test does what I stated initially: start a backstage server, let it run for a while, and then query its catalog API over HTTP to see if it contains the records I expect. Let's break down how this is accomplished.

Controlling the Server

The first thing we do is to start a Backstage server with the following declaration:

  let catalog: CatalogApi;

  beforeEach(function* () {
    catalog = yield createBackstage();
  });

This deceptively small piece of code bears quite a bit of weight. It starts a Backstage server and returns a reference to a Catalog API pointing to that server so that we can query its state.

createBackstage starts a completely fresh backstage server, with a completely new database for every single test case. The Backstage logs are persisted on a per-test basis in a special tests/logs directory to aid in debugging what went wrong whenever there is a failure.

This way, you achieve isolation, foster transparency, and avoid hidden dependencies between your tests. It may seem like a tall order, but this is where structured concurrency, and in particular, the application of its principles to javascript can help out.

Structured concurrency ties the lifetime of the Backstage process to the test. If the test passes, Backstage is shut down. If the test fails, Backstage is shut down. If the test errors or times out, Backstage is shut down. In other words, no matter the test's outcome, the Backstage server process will always be terminated. And importantly, at no point did we have to tell it to do so via an afterEach hook or some other explicit teardown mechanism.

These process guarantees give you incredible flexibility by allowing you to embed the Backstage server anywhere in your test case and not worry about how it is cleaned up.

Assertions Made Simple

Now that the server process is running, how can you make reliable assertions against it? Ideally, assertions should be as simple, fast, and predictable as if you were unit testing a pure synchronous function.

The problem, of course, is that, unlike a pure function, a server is a kinetic mass of state that is in continuous flux. Nevertheless, if you wait long enough, its internal motion will carry it to the point where we will reliably observe the state you expect to see, which is to the presence of the artist lookup service in the Catalog. Now is when the "wait a while" comes into play. The main question is, how long should the test wait?

Believe it or not, the answer is "long enough."

You could try an explicit wait via setTimeout(), but that is a well-documented antipattern. Alternatively, you could get some callback from Backstage that fired after each ingestion cycle was complete, but that adds an internal API used only for testing and means that we can no longer treat our server as a black box.

It turns out the theory of eventual consistency and its concept of "state convergence" can determine how long is long enough with relative ease.

In a regular synchronous assertion, there is no delay between cause and effect:

Diagram of an assertion at the right time

The effect of any action is immediately observable, and so when an assertion runs, it can always see that effect, making it simple and reliable.

However, in a highly asynchronous environment such as a Backstage server, you will likely try to run an assertion before an action's effects have yet to settle and become externally observable.

An assertion yielding a false negative

This would result in a false negative, where if you'd just had the patience to wait just a wee bit longer, you could have observed the effect, but instead, you made your assertion too soon, and the test failed.

Here is when eventual consistency comes to the rescue. If you assume that the server state is not immediately available, but is instead only in the process of converging onto some expected state, then instead of making a single observation at the right time, you continually make the same observation again and again until it eventually becomes true.

Converging assertion

Convergence guarantees that if the state you expect can be observed, it will be observed and that you'll wait just long enough to observe it and no longer. If there is something wrong, and the state is never achieved, then the test case will time out, and you'll be presented with the last failed assertion.

That may seem like a lot of conceptual overhead, but it's all put on the table so that when it comes to writing and reading test cases, your assertions are as simple as though your Backstage server were just another object.

it.eventually('ingests the artist lookup component ', function*() {
    const component = yield catalog.getEntityByRef('component:artist-lookup');
    expect(component).toMatchObject({
      metadata: {
        name: 'artist-lookup',
        description: "Artist Lookup",
        tags: ["java", "data"],
      },
      spec: {
        type: 'service',
        owner: 'team-a',
        system: 'artist-engagement-portal'
      }
    });
  });

Except for the eventually marker, that's the kind of assertion you would expect if you were testing a pure function.

The Dividend

The upshot of all this is that you now have a test suite that can answer the most critical question and crystal clear in how those questions are expressed. Not only that, but it does so reliably and without any need to account for the timing of internal server processes.

As a result, we have converted our Backstage server from an "environment" that needs to be set up, torn down, and carefully managed throughout our suite into a lightweight resource to be instantiated and tested according to the usual fashion.

If you want to dig deeper into this testing technique, check out the complete source code in Github. If you have any questions, ask away in our discord community!

Up Next

Not all Backstage ingestion is done with simple YAML files as the Catalog. Often it can be a complicated process involving integration with many third-party services. Building on the test suite in this article, we'll show how you can test these integrations without resorting to fixtures, mocking, stubbing, or other hacks.

Backstage tutorial: plugin integration tests with Cypress + Interactors

Jorge Lainfiesta — Thu, 03 Mar 2022 05:00:00 GMT

Plugins are the most common way to extend the power of Backstage. A plugin most often includes a UI built using @backstage/core-components and other Material UI components.

While Material UI makes it quick and easy to put together a good-looking UI, it also poses challenges when testing.

Consider this button:

<Button variant="contained" size="large" color="primary">
  =
</Button>

It generates the following output:

<button class="MuiButtonBase-root-201 MuiButton-root-216 MuiButton-contained-224 MuiButton-containedPrimary-225 MuiButton-containedSizeLarge-234 MuiButton-sizeLarge-236" tabindex="0" type="button" aria-label="=">
  <span class="MuiButton-label-217">=</span>
  <span class="MuiTouchRipple-root-304"></span>
</button>

That's a lot of generated classes. How will you know which selectors to use for your test actions and assertions? Usually you'd have to guess and try to craft selectors for each MUI component you use every time you use one.

Happily you don't have to worry about this anymore! The Frontside team has gone through the MUI components and created "Interactors" like the following that you can use in your tests:

import { Button } from '@interactors/material-ui';
//...
Button('=').exists()

This tutorial will show you how to set up integration tests for your Backstage plugin using Cypress and Interactors. In it you’ll learn how to:

Set up Cypress and Interactors
Assert with Interactors
Act with Interactors
Extend existing Interactors
Write your own Interactor
Use other Material UI Interactors

What you'll be testing

I created a Backstage plugin calculator with Material UI to use as an example for this tutorial. It features a 'classic calculator' with a button-based UI and a 'text-based calculator' which relies on a text field for input:

A gif showing the calculator demo

The calculator doesn't do anything exciting, so we won't go over the implementation details. You can have a look at the whole code in this repository.

Set up Cypress and Interactors

By default, Backstage sets up new plugins with Jest. Jest can help you with unit tests, but integration tests have significant performance problems due to the DOM virtualization.

Thus, I recommend using Cypress for your plugin's integration tests, which Backstage uses as well to test the instance as a whole.

I won't cover Cypress in much detail because you probably already have it set up for Backstage. All you need to do is apply those configurations and practices to your plugins too.

There's actually only two Interactors-specific settings to consider. First, make sure you're importing @interactors/with-cypress in your plugin's cypress/support/index.ts file. Secondly, adjust your eslint settings to accommodate Interactor's assertions in your plugin's cypress directory:

{
  "plugins": ["cypress"],
  "extends": ["plugin:cypress/recommended"],
  "rules": {
    "jest/expect-expect": [
      "error",
      {
        "assertFunctionNames": ["cy.expect"]
      }
    ],
    "new-cap": "off"
  }
}

Next, add the Material UI Interactors to your plugin. You know the drill:

yarn add --dev @interactors/material-ui

You can use Interactors with Jest in this step if you want, but because we'll be using them with Cypress, we need to install specific bindings:

yarn add --dev @interactors/with-cypress

Assert with Interactors

I'll use the Calculator plugin I created as an example for this tutorial. You can clone my repo and work from testless-calculator branch if you want to follow along.

The first test we'll write is to assert that the plugin's entry page renders when someone visits the plugin's URL:

import { Heading, Tab } from '@interactors/material-ui';

describe('The calculator plugin', () => {
  beforeEach(() => cy.visit('/calculator'));

  it('should render simple calculator by default', () => {
    cy.expect([
      Heading('Classic Calculator').exists(),
      Tab('CLASSIC CALCULATOR').is({ active: true }),
    ]);
  });
});

cy.expect is one of Interactors bindings for Cypress and it's used for assertions (the ther binding is explained next section). In the test above, you're telling Cypress to check if a MUI Heading with the text "Classic Calculator" exists and if a MUI Tab with the label "CLASSIC CALCULATOR" is active.

Screenshot of tests passing

Act with Interactors

Interactors enforce an AAA (Arrange-Act-Assert) pattern to provide helpful errors. So far, you've seen the function used for assertions (cy.expect). Now let's take a look at actions, well, in action with cy.do.

The second test you'll write will confirm that the inner navigation of the plugin works correctly. To do that, you'll have Cypress click on a tab that takes the user to another page and verify if the UI reflects that state:

  it('should change to input calculator with tab', () => {
    cy.do([
      Tab('TEXT CALCULATOR').click(),
    ]);
    cy.expect([
      Heading('Text-based Calculator').exists(),
      Tab('TEXT CALCULATOR').is({ active: true }),
    ]);
  });

cy.do is the other Interactors binding for Cypress, and it lets you perform actions. As you can see, you use the same Interactors to act and assert.

Extend existing Interactors

The classic calculator from the example also has some interactions that need to be tested. A recurring element that we'll need to check is the calculator's results box, so let's write an Interactor to target it:

import { HTML } from '@interactors/material-ui';

const CalculatorResult = HTML.extend<HTMLParagraphElement>(
  'Calculator result',
).selector('.calculator-results');

In the code above you're creating a new Interactor called "Calculator result", which extends the base HTML Interactor and specifies its selector as .calculator-result. (You can organize your Interactors in dedicated directories and files, but for simplicity I added this interactor on top of the spec file.)

Let's use the new CalculatorResult Interactor to check that clicking on the digit buttons results in their corresponding value appearing in the results panel:

it('should show inputted numbers in the result box', () => {
  cy.do([
    Button('0').click(),
    Button('1').click(),
    Button('2').click(),
    Button('3').click()
  ]);
  cy.expect([
    CalculatorResult().has({ text: '123' })
  ]);
})

You're checking that the calculator result corresponds to the digits pressed. It's worth noticing that the CalculatorResult() needs to match a single element or it'll throw an exception.

In this example, there's no way two results will be present on the same page, so if that did happen you'd certainly want your test to fail. You can specify filters and matchers to fine-tune your Interactor selection so it only picks up one element.

Write your own Interactor

The classic calculator has several cases to be tested. Does addition and substraction work correctly? What about the sequence of operations? If I wanted to test an operation like 101+23-7, I'd need to write a test like the following:

it('should add two numbers correctly', () => {
  cy.do([
    Button('1').click(),
    Button('0').click(),
    Button('1').click(),
    Button('+').click(),
    Button('2').click(),
    Button('3').click(),
    Button('-').click(),
    Button('7').click(),
    Button('=').click(),
  ]);
  cy.expect([
    CalculatorResult().has({ text: '117' })
  ]);
});

As you can imagine, testing more complex operations can be burdensome with all the manual button clicking. Instead, you can write an interactor for the classic calculator that performs the button clicking for you:

const ClassicCalculator = createInteractor('Calculator')
  .selector('.classic-calculator')
  .actions({
    async inputDigits(calculator, digits: string) {
      for (const digit of digits) {
        await calculator.find(Button(digit)).click()
      }
    }
  });

In the code above, you're defining an Interactor called "Calculator" and specifying an action for it. The inputDigits action will receive this calculator interactor instance as the first argument with a digits parameter. For every digit, you're finding a Button interactor with the specified digit within the calculator Interactor and clicking it.

Using your ClassicCalculator Interactor, you can write the previous test example as:

it('should add two numbers correctly with Interactor', () => {
  cy.do([
    ClassicCalculator().inputDigits('101+23-7=')
  ]);
  cy.expect([
    CalculatorResult().has({ text: '117' })
  ]);
});

Other Material UI Interactors

@interactors/material-ui comes packed with 30+ Interactors for Material UI components. You can choose whichever interactor suits your UI from the detailed MUI Interactors API.

To illustrate how useful these are, let’s use another Interactor with a test for the other tab of my calculator plugin. Instead of a button-based input, the "Text-based calculator" accepts a math expression from the user and prints the result to the result box. Thus, I can use the Material UI TextField Interactor as follows:

import { Button, TextField } from '@interactors/material-ui';

describe('The text calculator', () => {
  beforeEach(() => cy.visit('/calculator/text'));

  it('should render initail state', () => {
    cy.expect([
      TextField('Math expression').exists(),
      Button('=').exists(),
      CalculatorResult().has({ text: '0' })
    ])
  });

  it('should solve an expression', () => {
    cy.do([
      TextField('Math expression').fillIn('101+23'),
      Button('=').click(),
    ]);
    cy.expect([
      CalculatorResult().has({ text: '124' })
    ]);
  });
});

If you run the test suite based on Cypress and Interactors, it should look like this:

A gif with tests executing on cypress

Conclusion

Using Interactors in your Backstage integration tests can help you save time and development efforts. Try them out, and if you need any help, reach out via our Discord server and ask away!

Announcing GraphGen: the search for the graph factory

Charles Lowell — Mon, 21 Feb 2022 05:00:00 GMT

From the very beginning of my career until the present day I have relied on fake data. That may sound odd to the uninitiated—especially in a world where that seems overrun with fake data—but it’s true for me and every other developer I know. We couldn’t do what we do—build reliable applications that work every time—without high-quality fake data.

The first such set I can remember using was tinydata.sql, an SQL file that contained an aggressively curated list of INSERT statements that reconstructed a complete and realistic dataset from scratch. “Tiny Data” was quite revered by the developers in the team because it was so critical to both testing and developing apps. Indeed, I had so much respect for Tiny Data that I would often sing to it in the tune of Elton John’s “Tiny Dancer.” Not only did it allow you to see fake high-fidelity receipts and product data for the point of sale application we were writing, but it also let you write tests that could rely on certain records always being present. You were in big trouble if you made a change to the application that was incompatible with Tiny Data because it would instantly disrupt the workflows of more than thirty developers. They would not be happy about it at all.

There has been a lot of progress since those days. We now recognize the value of high-quality fake data, and we intentionally place it at the center of our development practice. Instead of static hand-curated datasets we use factories, and instead of obviously fake names like “Test User 1” we use tools like faker.js to generate random yet realistic values. However, despite these improvements, other problems remain as vexing now as they were back then.

Though I didn’t know the name for it at the time, Tiny Data was what we now call a fixture. And while we all did our very best to make sure that it was always in sync with the app we were developing, we somehow managed to break it pretty often. Invariably the breakage had to do with relationships between rows—or in the parlance of SQL, maintaining the foreign keys. For example, if you wanted to add a receipt, then it wasn’t just enough to add a row to the receipts table. You also needed to make sure and add a corresponding product, store, and clerk. If you didn’t, otherwise it wouldn’t truly be valid: it might work just fine in the context of a daily receipt listing, but then break if you tried to navigate through to the receipt details page.

Fast forward twenty years, and it is still the relationship between records that causes the most problems with fake data. Isn’t it odd that while we’ve gotten good at generating realistic data structures, we still run into problems when trying to relate those data structures with each other? Why is that so? And what can we do about it?

Consider for a moment the critical innovation of tools like faker.js—you use a randomly selected value and yet it feels real. You reach your hand into a bag and draw out a name, an address, whatever you need. But what if we could reach our hand into a bag and draw out a fully formed and realistic graph? In other words, what if the network itself was the fake data and not the individual nodes within it? What would that look like?

This was the question we decided to try and answer in late 2021. The result was @frontside/graphgen (https://github.com/thefrontside/graphgen). Like faker.js, it uses a probabilistic approach to generating values—except now the values generated are graphs instead of names or addresses.

Much to our surprise, even in its most rudimentary implementation the tool was instantly powerful. Using a single generated graph, we were able to drive a GitHub simulator, an LDAP simulator, and a custom GraphQL simulator, all of which reported a state of the world consistent with each other. The same user data could be found in LDAP, GitHub, and the custom API.

Most posts announcing a new tool do so at the end of a long journey after which a finished product is presented with a bow on top for general availability. This is not that, because that’s not how we do things at Frontside. Instead, we’re announcing graphgen at the beginning of its creation. We don’t know what form it will take eventually, and there are undoubtedly many challenges to overcome along the way. But we are sure that we’ll create a future where you can summon—in the blink of an eye—a fake dataset nearly indistinguishable from production.

Tutorial: How to use your company’s component library with Backstage

Taras Mankovski — Mon, 14 Feb 2022 05:00:00 GMT

One of the first questions that a developer starting a Backstage project has to answer is how to make their developer portal match their company’s corporate style. There are several ways of doing this, but I recommend using an approach that will allow developers to use as much of the Backstage UI as possible so that everyone gets the most value from Backstage. The further the developer portal is from the beaten path, the fewer Backstage elements the development team will have access to.

The approach that retains the most value from Backstage is using a custom Material UI theme which allows developers to use all of the existing Backstage plugins. If specific components need to be modified, there is the option of overriding Backstage and Material UI styles. Admittedly, this approach can take a fair bit of tweaking and can be somewhat tedious, but will likely result in being able to use all of the existing and future Backstage plugins.

If using the UI components provided by Backstage plugins is less important than using your internal component library, there are still good reasons to maintain access to their data. In addition to Material UI components, the Backstage frontend framework provides a data fetching API for retrieving data from the backend. Even if you’re not using the UI components of Backstage plugins, you can continue to benefit from them by leveraging their data API.

In this tutorial, I will illustrate how to replace Material UI with the Fluent UI component library, then use the Catalog Client API and FluentUI’s DetailList to display a list of catalog entities from Backstage's backend. Let’s get started!

Create a Backstage App

If you don’t already have an instance of a Backstage app, create one by running:

npx @backstage/create-app

For this tutorial, you can select SQLite as your backend database as it will save you the hassle of having to run a local instance of Postgres.

Clean up your app

Before we start adding our theme, we need to do a bit of clean up. Since we’re not going to be using Material UI, there is no need to keep its dependencies. We’ll start by removing those packages:

yarn workspace app remove @backstage/theme @material-ui/core @material-ui/icons

You can also go ahead and delete everything in the packages/app/src/components/ directory and create a simple Home component that will be our new Home page (without all the Backstage frontend elements):

// packages/app/src/components/Home.tsx

import React from 'react';

export const Home = () => {
  return <p>Home</p>;
}

Next, you need to remove all the routes and plugins and add the new Home component to your app. There is a lot to remove so I’ll show you what your App.tsx in ./packages/app/src/ should look like:

import React from 'react';
import { Route } from 'react-router';

import { apis } from './apis';
import { createApp } from '@backstage/app-defaults';
import { FlatRoutes } from '@backstage/core-app-api';

import { Home } from './components/Home';

const app = createApp({
  apis,
});

const AppProvider = app.getProvider();
const AppRouter = app.getRouter();

const routes = (
  <FlatRoutes>
    <Route path="/" element={<Home />} />
  </FlatRoutes>
);

const App = () => (
  <AppProvider>
    <AppRouter>
      {routes}
    </AppRouter>
  </AppProvider>
);

export default App;

Install your component library

Next, install the @fluentui/react component library to your frontend by running:

yarn workspace app add @fluentui/react

This will allow you to use components from the library in the frontend application. We can now modify Home.tsx to display a component from @fluentui/react:

import React from 'react';
+ import { PrimaryButton } from '@fluentui/react';

export const Home = () => {
-  return <p>Home</p>;
+  return <PrimaryButton>Press me!</PrimaryButton>;
}

If you run yarn start, you should be able to see a button with the default Fluent UI style that says “Press me!”.

Change the theme provider

By default, Backstage’s createApp uses a Material UI theme provider that adds Backstage’s specific styles. We want to replace this default theme provider with the one from your component library. To do this, you need to modify App.tsx:

- import React from 'react';
+ import React, { ComponentType } from 'react';
+ import { ThemeProvider, PartialTheme } from '@fluentui/react';
...
+ const myTheme: PartialTheme = {
+   semanticColors: {
+     primaryButtonBackground: 'red',
+     primaryButtonText: 'white',
+   },
+ }
+ 
+ const MyThemeProvider: ComponentType<{}> = ({ children }) => {
+   return (<ThemeProvider theme={myTheme}>{children}</ThemeProvider>);
+ }
+
const app = createApp({
  apis,
+  components: {
+    ThemeProvider: MyThemeProvider,
+  }
})

Once you make these changes, your UI is now using your component library. Be aware that if you render any components that come with Backstage, they will not work because the default Backstage theme provider will be missing.

Use Backstage Catalog API to fetch data and display it in a table

The work that you did so far was to add a new theme to the Backstage component. You did this by modifying the frontend app. Other teams sometimes take a different approach where they create an entirely new Create React App project and import that to their Backstage app. Starting a new project may seem appealing, but in practice you’d be giving up more features than you realize. One of the most important lost features is the ability to call data APIs provided by the Backstage framework.

If your connection to the backend is intact, you can still use the data fetching API provided by Backstage. The Backstage UI components you deleted earlier already had data fetching implemented, which means the Home component you created is missing that functionality. I will now show how to use the data API to fetch data from the Backstage backend and display it in a DetailList component.

Make catalog API available in your application

First, install @backstage/catalog-client:

yarn workspace app add @backstage/catalog-client

You can make the Catalog API available in your application by adding an API factory to the apis.ts file in your package/app/src/ directory:

  createApiFactory,
+   discoveryApiRef
} from '@backstage/core-plugin-api';
+ import { catalogApiRef } from '@backstage/plugin-catalog-react';
+ import { CatalogClient } from '@backstage/catalog-client';

export const apis: AnyApiFactory[] = [
+   createApiFactory({
+     api: catalogApiRef,
+     deps: { discoveryApi: discoveryApiRef },
+     factory: ({ discoveryApi }) => new CatalogClient({ discoveryApi })
+   }),
  createApiFactory({

Let’s unpack what you see above. You will use the Catalog API reference (which comes from @backstage/plugin-catalog-react) to look up the API in your components when using the useApi hook. The AnyApiFactory factory controls how the Catalog Client is created. CatalogClient needs DiscoveryApiRef, which provides the factory access to the DiscoveryApi. The DiscoveryApi is used to find the URL where the Backstage Catalog API can be found. This piping will make it easy to call the API in future steps.

Use `useApi` hook to get the CatalogApi

Let’s get rid of the PrimaryButton from earlier and make the following additional changes in your Home component:

- import { PrimaryButton } from '@fluentui/react';
+ import { catalogApiRef } from '@backstage/plugin-catalog-react';
+ import { useApi } from '@backstage/core-plugin-api';

export const Home = () => {
-   return <PrimaryButton>Press me!</PrimaryButton>
+   const catalogApi = useApi(catalogApiRef);
}

This provides your component with access to reference to the Catalog API. As a result, we can call this API to retrieve data from the Catalog API.

Use `useAsync` hook to call the CatalogApi

To call the CatalogApi, we need to call catalogApi.getEntities function. This function needs to be called with our component renders. We can use the useAsync hook to call catalogApi.getEntities:

+ import useAsync from 'react-use/lib/useAsync';

export const Home = () => {
  const catalogApi = useApi(catalogApiRef);
+   const {
+     value = { items: [] },
+     // loading,
+     // error,
+   } = useAsync(() =>
+     catalogApi.getEntities({
+       filter: {
+         kind: 'Component',
+       },
+     }),
+   );
}

We could implement the same thing with useEffect and useState but it’s a lot more work. useAsync automatically gives us loading and error values so we don’t need to handle those situations manually.

Use `useMemo` hook to convert result from CatalogApi into items for `DetailsList`

FluentUI’s DetailList needs to convert results from the CatalogApi. The data needs to be in a specific format so that we can display it in the table:

- import React from 'react';
+ import React, { useMemo } from 'react';

export const Home = () => {
  ...
+   const items = useMemo(
+     () =>
+       value.items.map(item => ({
+         key: item.metadata.uid,
+         name: item.metadata.name,
+         description: item.metadata.description,
+       })),
+     [value],
+   );
}

useMemo allows us to covert the response from a CatalogApi response format to DetailList format and cache the result. The caching part is important because without it the list will re-render every time anything in that component changes.

Render `DetailList` with items

The final step is to import DetailList from @fluentui/react and render it. Making this last change should generate a list of items from the catalog using the Catalog API and rendered in the components from FluentUI library:

+ import { DetailsList } from '@fluentui/react';

export const Home = () => {
  ...
+   return <DetailsList items={items} />;
}

To check to make sure these new changes are working properly, run yarn dev (you’ll need the backend to be running too).

This whole process is a bit involved but doing it this way saves you a lot of trouble in the long run by allowing you to use the data fetching APIs that come with Backstage plugins.

Conclusion

In this tutorial, I showed you how to clean up the frontend application in preparation for using your own component library, install a component library other than Material UI, override the theme provider to use your component library’s theme provider, and use Backstage’s data fetching API provided by the Catalog API plugin.

This is the “right” way to use your company’s component library because it allows you to use it without completely eliminating all of the benefits that the Backstage framework provides in the frontend application. You might not be able to use the components that Backstage plugins provide because they are tied to Material UI, but you can still use the data fetching APIs that idiomatic Backstage plugins include.

I hope you enjoyed this tutorial and learned something about Backstage. You can find me on Twitter, GitHub, or via email - tarasm@frontside.com. All of the code from this tutorial can be found in https://github.com/taras/backstage-custom-frontend

When to use SemVer or CalVer: project type considerations

Jacob Bolda — Wed, 09 Feb 2022 05:00:00 GMT

Reaching meaningful versions is more than setting a number within a package's main file (e.g., package.json, pubspec.yaml, etc.). Versioning aims to build up quality metadata that humans and computers can understand. With this metadata, you can understand the impact a change will have downstream and communicate with users more effectively.

The two most common versioning schemes—SemVer (semantic versioning) and CalVer (calendar versioning)—can help you build the desired aggregated understanding. But going with either of them is not a boolean choice but a spectrum whose adherence specifications you must outline depending on your project's needs and circumstances.

For instance, strict SemVer tends to have the most value in libraries because it is easy to offer rationales for changes and tie them to version numbers. But as codebases become larger and more multifaceted or marketing gets involved, the decision to cut a major version release becomes less about breaking changes and more driven by external factors.

On the other hand, projects with large, shifting codebases tend to prefer using the year for the major version. This works particularly well if there is a Long Term Support (LTS) and maintenance release schedule, as in the case of Ubuntu. It might also lend itself well as a scheme for marketing purposes.

In this article, I'll outline the notions behind SemVer and CalVer and then propose considerations for which versioning scheme to use depending on whether your project is a library, an app, a service that expects breakage, or an additive-only service.

What is SemVer?

SemVer's fundamental rules are simple:

Given a version number MAJOR.MINOR.PATCH, increment the:
1. MAJOR version when you make incompatible API changes,
2. MINOR version when you add functionality in a backward-compatible manner, and
3. PATCH version when you make backward-compatible bug fixes.

However, it is hard to track which version we are dealing with in a living codebase with dozens of developers continuously contributing changes. Thankfully, teams can nowadays set up tools like changesets or covector (for monorepos) to automatically aggregate versions. That means that the team must clearly define bump strategies so that all developers have a consistent understanding of the different kinds of changes.

Furthermore, adhering to SemVer's strict definition is not always convenient. For instance, a project may only rely on a MAJOR.MINOR and group all bug fixes and features into a minor release. Or a team may use MAJOR.MINOR.PATCH internally when publishing and releasing, but only publicly release the MAJOR version for consumption. The critical consideration is that the strategy employed is communicated clearly and then consistently used.

What is CalVer?

While SemVer proposes rules that determine the version number, CalVer invites its adopters to define their scheme, considering how time impacts their project. The scheme structure is similar, MAJOR.MINOR.MICRO, but what each segment means is more fluid. In most cases, the MAJOR number is linked to the year of the release. For MINOR and MICRO, CalVer adopters use ad-hoc strategies and combine notions from SemVer.

CalVer has natural synergies with marketing schedules, as they can be tied to the shared time-based milestones.

Versioning considerations by project nature

One way to resolve the question of which versioning model to adopt is to let the nature of your project decide for you.

Libraries

A library lends itself well to versioning using SemVer. Smaller scoped libraries with well-understood changes and impacts can more naturally adopt SemVer in a relatively strict manner. But as the velocity of code (the number of commits in a given period) increases and inter-dependencies build up, decisions around major and minor become more involved, and you might choose to loosen the stringent requirements.

As code velocity increases, it makes sense to group commits into larger releases. But as this progression happens, it becomes more challenging to release a package separate from other changes that have been committed to the monorepo. In light of this, SemVer allows for dependencies within a monorepo to calculate a bump for related packages in response to a specific code change.

Web apps / mobile apps

The value and schema of a version number have different expectations in web or mobile apps because users will rarely be concerned about it. Thus the choice between SemVer and CalVer lies less in the nature of the codebase nature or its users but rather in external influences. A user may not see internal shifts in the code, but they will notice an extensive design refresh, which is often tied to marketing efforts.

The different expectation doesn't necessarily dictate one option over the other, as either would allow communication between the user and support and the user/support to the developer. Instead, the release cadence and marketing schedules will most likely dictate the versioning scheme. The app may have a plan that best matches releases at CalVer’s specific time intervals, which is often seen in Tax prep software offerings. If there is no synergy with other departments, following SemVer is probably simpler for developers.

Services that expect breakages

Services such as REST are designed with the expectation of breaking changes and it is common for REST APIs to use the major version within the URL. The incremental changes may be internally considered a minor and a patch version, but the external version may only be a major version. The SemVer scheme seems to lend itself well to services, but maintaining many service versions increases complexity.

Additive-only Services

An additive-only schema paradigm (where every change can only add to the schema and not cause any breaking changes) has recently come to the forefront. This paradigm devalues major, minor, and patch use, as every release should theoretically be minor. However, in reality there could be breaking changes, but such changes would be elegantly handled through deprecations. As the schema is adjusted, fields can be marked as deprecated. These fields still allow use but will be monitored, and completely removed from the schema when their activity falls below a threshold (while this is technically a breaking change, messaging about a major change does not need to be created to prepare developers to make these changes on a timed schedule).

In light of this rhythm, a case could be made for CalVer. Deprecated fields below the activity threshold could be removed on an annual cadence and the major version bumped to a new year. Every other change only needs to communicate the addition to the schema, expressed by bumping the second number or minor version (there may not be value in using a patch version in this situation).

Conclusion and Recommendations

In general, developers find it easier to work with a SemVer scheme because it's more intuitive given that it's closer to the code. But as projects grow, more considerations have to be made to define a versioning scheme that communicates effectively with users.

It is also not uncommon for projects to change their versioning scheme as they evolve. A project started using strict SemVer doesn’t have stick with that approach; don't be afraid of revising that decision further down the line if it's causing churn or confusion for developers, users, or other teams.

But no matter what you decide, explicit definitions of your versioning scheme helps developers and users navigate changes and expectations over time. Be sure that these norms are written down and accessible for all stakeholders.

What is DX anyway: a chronological overview

Jorge Lainfiesta — Wed, 26 Jan 2022 05:00:00 GMT

It was bound to happen: after UX (user experience) and CX (customer experience) it was only a matter of time before DX (Developer Experience) took center stage. Nowadays it's in the marketing copy of every library and product aimed at developers. But what does DX mean?

As it turns out, the definition of DX has evolved in important ways over the past decade. This month in the DX Newsletter, we’ll review three trends found in academic and industry articles to set the stage for a broader conversation.

1. The first traces of DX

The combination of “developer” and “experience” yields a jumble of ambiguous concepts throughout the years. For instance, most results point to the seniority of a developer.

But as we understand it today, “Developer Experience” was used first in 2012 in a paper by Finnish Ph.D. candidates published in the renowned technical journal IEEE.

The authors built a conceptual framework for Developer Experience, drawing concepts from psychology and UX practice:

A diagram depecting three dimensions of DX: cognition, affect, and connation

Adapted from Figure 1: Developer Experience: Conceptual Framework (*Fagerholm F. and Münch J., 2012).

2. The code-centered era of DX

Subsequently, most references to DX before 2021 were mainly concerned with code, with API ergonomics, documentation, and a support community taking a privileged place.

For example, as Red Hat argued in 2020, good documentation and a welcoming community are “essential parts” of DX and will help you decide if it’s even worth trying out an API, no matter how “ergonomic” its creators deem it to be.

For teams creating tools, services, or products for developers, Sam Jarman wrote an ever-green DX guide back in 2017 with practical considerations to improve any API’s documentation and distribution.

3. Holistic DX: the present and future

Code is fundamental in DX, but developers spend only around a third of their time coding. In 2021, we’ve come full circle from 2012’s psychology-based definition of Developer Experience. Now the attention is back to the developer as a whole: tools are still a focus, but so are factors like psychological safety.

Jean Yang, CEO of Akita Software and former professor of Computer Science at Carnegie Mellon University, proposes that DX has two aspects: one is abstracting complexity away, and the other (less explored purpose) is helping developers navigate complexity. Developers face messy and complex problems in their daily work, and can rarely swap their systems for new abstract “dev-friendly” tools.

In sum, improving the Developer Experience for any team is a holistic endeavor that encompasses technology, working practices, and organizational enablement. As Microsoft explained in a review of four industry case studies, companies undertake multi-year plans to improve their DX to achieve more productivity and better retention rates.

Originally published in our DX Newsletter. Subscribe below!

Tutorial: Easier onboarding with Backstage and VSCode dev containers

Min Kim — Mon, 24 Jan 2022 05:00:00 GMT

An often underrated aspect of developer onboarding is how quickly a new contributor can go from cloning the source code to running the project on their machine. An onboarding experience full of friction can be discouraging and leave a poor impression, while a smooth one is valuable for new and seasoned contributors alike.

However, onboarding can be especially challenging for Backstage because it is a full-stack application. Developing a Backstage-based Developer Portal requires having a recent version of Node.js, Yarn, and PostgreSQL running on the developer's machine. These are relatively easy to install on macOS and Linux but can be challenging on Windows, especially for someone new to the Node.js ecosystem.

At Frontside, we've had success improving onboarding by using dev containers, which spins up a docker-based local environment with all the dependencies that a project needs.

Dev containers has a CLI called devcontainer-cli you can use to open the container from the command line. It will open in VSCode, which uses Remote Containers Extension to create a smooth integration.

For the most part, running dev containers inside VSCode feels like the application is running on your machine. Once you've added dev containers to your project, there’s a clear pathway to adopting GitHub Codespaces, which makes onboarding even easier.

This blog post will explain setting up dev containers for Backstage projects as a first step to create a streamlined onboarding for developers.

Requirements

Before you get started, make sure you have the following:

Visual Studio Code
Remote Containers Extension for VSCode
Docker Desktop
Docker Compose (if you're running Linux)

Create a Backstage App

To spin up a dev container, you first need a project to work from. You can instantiate a Backstage app using @backstage/create-app:

npx @backstage/create-app

For this tutorial, you'll be using PostgreSQL:

backstage-create-app

Once the app is done installing, open the new app's workspace in visual studio code and set up a dev container.

Opening a Project inside a dev container

If you have the Remote Containers extension installed, you should be able to access its commands. Press F1 and select Remote-Containers: Open Folder in Container...:

open-folder-container

VSCode will detect that you do not have dev container configurations yet and prompt you to select a predefined container. You can go ahead and select any of them, as you'll be replacing all of those configurations later.

Once you make your selection, VSCode will create configuration files in the .devcontainer directory, build the container according to those files, and launch your workspace in the container. (You can go to your Docker Desktop dashboard to confirm that the new container is running.)

If you change your dev container configuration, you can run Remote-Containers: Rebuild Container. And when you are ready to exit out of your dev container, run Remote-Containers: Reopen Folder Locally.

Now that you know how to launch and exit out of dev containers, you need to add configurations to run Backstage.

Docker Compose - Postgres

If you try to start up Backstage now, you'll get an error in the terminal about there being no database to connect to. This is because you're not running an instance of Postgres yet, so you should start there. At the root of your project workspace create a docker-compose.yaml file:

version: "3.8"

services:
  my_postgres_db:
    image: postgres
    environment:
      POSTGRES_USER: postgres_username
      POSTGRES_PASSWORD: postgres_password

You're creating a my_postgres_db service with the official postgres docker image and providing it a username and password to bootstrap a Postgres account for local development.

Docker Compose - dev container

Next, create a service for your dev container (this will replace the one that VSCode automatically generated earlier).

services:
  my_postgres_db:
    ...
  my_devcontainer:
    image: mcr.microsoft.com/vscode/devcontainers/typescript-node:14
    command: /bin/sh -c "while sleep 1000; do:; done"
    volumes:
      - .:/workspace
    environment:
      POSTGRES_HOST: my_postgres_db
      POSTGRES_USER: postgres_username
      POSTGRES_PASSWORD: postgres_password

For your dev container image, you’re using one of the docker images created by Microsoft and running a shell command to prevent the container from exiting on its own as suggested in the devcontainer docs.

For a container to have all of the files of the backstage app, you need to add volumes: .:/workspace as shown in the previous snippet. This will take the current directory (relative to the docker-compose file) and copy its contents to your container's /workspace directory.

You're also passing in environment variables to access the local instance of Postgres. These should correspond to the values you specified when creating the my_postgres_db service. (Alternatively, you could provide these values as environment variables and pass them through the start script of your backstage backend.)

Rebuild `devcontainer`

Now you need to update your dev container configurations to use the docker-compose file you created:

// .devcontainers/devcontainer.json
{
  "dockerComposeFile": "../docker-compose.yaml",
  "service": "my_devcontainer",
  "workspaceFolder": "/workspace"
}

For these changes to be applied, you will need to rebuild your dev container by running either Remote-Containers: Rebuild Container or Rebuild and Reopen in Container, depending on if you returned to your local environment or if you're still in the first dev container you launched. You can go ahead and delete the other files that were created inside .devcontainer/.

Using Persistent Volumes

At this point, if you try to add a dependency or run yarn install, you may notice that it takes much longer than it usually does in your local environment. This is because when you create files in a docker container, it writes to the container's writable layer, which is not very fast. The standard practice (as suggested by docker docs) is to use volumes.

In your docker-compose file, add the three node_modules directories of your backstage app:

services:
  my_postgres_db:
    ...
  my_devcontainer:
    ...
    volumes:
      - .:/workspace
      - /workspace/node_modules
      - /workspace/packages/app/node_modules
      - /workspace/packages/backend/node_modules

This syntax can be a little confusing at first, and you may be wondering if and why .:/workspace omits node_modules. Although they are nested under the same volumes property, what they achieve is different. When you specify both a source and target path, docker-compose will treat that as a bind mount. But when you pass in only a target path, docker-compose will create an anonymous volume.

What's happening here is docker-compose first copies your backstage app to the /workspace directory of your dev container and then maps out the contents of node_modules to persistent volumes. You can read more about the differences between a bind mount and a volume here.

Once you complete these steps and rebuild and reopen your dev container, you should see a significant improvement when you run yarn install.

Dev containers are a first step

Now that you know how to set up dev containers optimized for Backstage development, you can make it easier for your teammates to contribute to your Developer Portal (regardless of whether they use macOS, Linux or Windows) by including the settings explained in this tutorial in your repository.

The setup we explained in this tutorial also allows people to opt-in (or not) to running dev containers, so these changes do not affect anybody with other local development preferences.

Dev containers are meant to lower barriers for those who cannot run the project locally. However, it's important to point out that even though dev containers are a convenient way to run the project, the DX of developing in a dev container is still inferior in some ways to running the project locally because it introduces the overhead of running Docker containers. With that said, we still believe that they are a handy tool and will drive the DX of your project in the right direction.

Simplified local development and testing with Auth0 simulation

Paul Cowan — Thu, 13 Jan 2022 05:00:00 GMT

Auth0 is an excellent service that lets you focus on your app instead of trying to stay up to date with the latest safe and secure authentication practices. However, what’s not so great is having to use Auth0 as part of your local development process or within end-to-end tests.

Recently I had a tough time using Auth0 while running tests that required authentication in a continuous integration environment with no access to the internet. The issue emerged because adding logical branches to my code targeting specific environments and stubbing functions at the test level leaves room for undesired effects.

There are many ways you can address the issue, but most of them are a disaster. Probably the most naïve solution is to write conditionals that check which environment the code is currently running against. Consider this example:

if (process.env.NODE_ENV !== 'production') {
  redirectToFakeAuthentication();
} else {
  auth.client.loginWithDefaultDirectory({
    username: email,
    password,
  // etc.

This kind of conditional forks your app by environment, leading to bugs escaping into production because developers and tests don't really assess the same app that ends up being deployed but a version catered for them only.

Unfortunately mocking or stubbing calls to Auth0 in a unit or end-to-end test never ends well either. It usually leads to an explosion of mocking code that you must maintain through every change. It may start small, as in the following snippet:

const mockConfig = (config) => {
  jest.doMock(
    '../auth_config.json',
    () => ({
      domain: 'test-domain.com',
      clientId: '123',
      ...config,
    }),
    { virtual: true }
  );
};

But further down the line, you'll find yourself writing mock JSON web tokens (JWTs) and specifying key algorithms, which means wasting precious development time. Before you know it, token checking code has crept into the tests (and who wants to maintain test helpers like this?):

export const verifyAuth0Token = async (token) => {
  return new Promise((resolve, reject) => {
    jwt.verify(token, getKey, { algorithms: ['RS256'] }, (err, decoded) => {
      if (err) {
        reject(err);
        return;
      }

      resolve(decoded);
    });
  });
};

Packages like mock-jwks help, but it’s still code that needs to be maintained, and as we all know, the less code we write, the fewer problems we have.

At last, we arrive at what seems to be the most popular solution but which is arguably the worst: to create phoney Auth0 accounts in multiple environments, as documented in this post. This is needless overhead. You'll have to keep all these environments or tenants in sync, and it does not cure the problem of running end-to-end tests behind a firewall with no internet.

Another downside of this approach is that all tests that use a particular phoney account will share state in different tests. Auth0 tenants are not scalable, which means we'd be moving headfirst into a maintainability nightmare.

Solution

However, the unworkable “solutions” presented above also point to the principle that will guide our workable one: namely that we should always strive to have the same code running in every environment. The only difference should be the configuration. For example, in production we might have this auth0 configuration:

{
  "domain": "myapp.eu.auth0.com",
  "clientId": "PMkiueyWaFdfsbAKXrIpVPmyBTFs4g5iq",
  "audience": "https://thefrontside.auth0.com/api/v1/",
  // etc.

And in non-production environments it might look like this:

{
  "domain": "localhost:4400",
  "clientId": "00000000000000000000000000000000",
  "audience": "https://your-audience/"
  // etc.

The only discernable difference is the domain field, which points to a localhost when running in non-production environments (such as your very own laptop or maybe even in a continuous integration environment (CI) that does not have access to the internet).

Imagine your very own Auth0 server running locally to configure specific scenarios while developing or running tests. What if we called it “auth0 simulator” and allowed you to create fake data without cluttering up your existing codebase?

Auth0 simulator

As chance would have it, such a beast exists as part of the simulacrum suite of tools, created to help simulate complex external bounded contexts such as LDAP or Auth0.

The @simulacrum/auth0-simulator package is your very own Auth0 simulator that eliminates the need to log into Auth0 while developing in a non-production environment. It is enormously helpful in local development and end-to-end testing – and even features ready-to-use integrations with Cypress!

Start your engines

To use it you'll first need to set up a simulation. You can either create your simulation using a GraphQL interface or through the @simulacrum/client JavaScript package:

async function setupClient({ url }) {
  let client = createClient(url);

  let simulation = await client.createSimulation('auth0', {
    options: {
      audience: 'https://your-audience/',
      scope: 'openid profile email offline_access',
      clientId: 'YOUR_AUTH0_CLIENT_ID',
    },
    services: {
      auth0: {
        port: 4400, // port for the auth0 service itself
      },
    },
  });

  console.log(`auth0 service running at ${simulation.services[0].url}`);
  let person = await client.given(simulation, 'person');
  console.log(
    `username = ${person.data.email} password = ${person.data.password}`
  );
}

Fake users are created through client.given with random values assigned by default. You can also supply the values if you want:

let person = yield client.given(simulation, 'person', {
  email: 'bob@gmail.com',
});

Local Development

The goal of @simulacrum/auth0-simulator is to behave just like the real Auth0 server does. That means that libraries like @auth0/auth0-react should not notice any difference so that you can keep using your regular code and expect it to work without any adjustment when you switch to the real deal. The following snippet would work just as well using the Auth0 simulator and the production environment.

import { useAuth0 } from "@auth0/auth0-react";

const LoginButton = () => {
  const { loginWithRedirect } = useAuth0();

  return <button onClick={() => loginWithRedirect()}>Log In</button>;
};

Testing

Having to redirect to login pages and enter user input when executing unit tests or end-to-end tests would be untenable. What’s especially useful about @simulacrum/auth0-simulator is that it makes local simulation instances available to test environments for end-to-end and unit tests without trips to Auth0.

End-to-end testing with Cypress

Simulacrum has a @simulacrum/auth0-cypress package that can silently get an authentication token from the Auth0 simulator often required to run end-to-end tests. In the following example, we'll create an Auth0 simulation instance within a test, enabling the app to run under the config we wish:

import type { Client, Scenario, Simulation } from '@simulacrum/client';
import { createClient } from '@simulacrum/client';
import auth0Config from '../../cypress.env.json';

describe('log in', () => {
  it('should get token without signing in', () => {
    cy.createSimulation(auth0Config)
      .visit('/')
      .contains('Log out')
      .should('not.exist')
      .given({
        email: 'bob@gmail.com'
      })
      .login()
      .visit('/')
      .contains('Log out')
      .logout();
  });
});

Unit tests

At present, the Auth0 simulator has no official helper for obtaining an access token from the simulation server. But most javascript Auth0 SDKs have helpers for a client login that you can abstract into a helper function and use in conjunction with @simulacrum/auth0-simulator in unit tests.

For instance, if you're using @auth0/auth0-spa-js, you could use getTokenSilently to obtain tokens:

import configJson from "../../src/auth_config.json";
import { Auth0Client } from '@auth0/auth0-spa-js';

const auth0Client = new Auth0Client({
  audience: configJson.audience,
  client_id: configJson.clientId,
  connection: 'Username-Password-Authentication',
  domain: configJson.domain,
  scope: 'openid profile email',
  cacheLocation: 'localstorage',
  useRefreshTokens: true
});

async function silentLogin({ currentUser }: {currentUser: string}) {
  await auth0Client.getTokenSilently({ currentUser });
});

or you could use loginWithDefaultDirectory if you're using 'auth0-js'.

Epilogue

Our Auth0 simulator @simulacrum/auth0-simulator makes Auth0’s “pain points” disappear by allowing developers to work locally and test their apps—without introducing fragmentation into their codebase, having to write cumbersome mocking code, or dealing with Auth0 accounts for different environments. If you are using Auth0 then you really should give here a spin.

Feel free join our Discord community, or contact us, for questions or help to get started!

Effection: for when async/await is not enough

Jonas Niklas — Tue, 26 Oct 2021 05:00:00 GMT

Everyone who has built a complex system in JavaScript has hit a critical moment where the illusion of control comes crashing down. Those are the moments when there are dozens (maybe hundreds) of concurrent processes running and it feels like you've lost control over them. Maybe a promise handler or callback executes even though it is no longer relevant and messes up the state of the system. Or perhaps an error disappears into the void or a socket is not closed when it should be.

The problem with concurrent processes in JavaScript is that it is fundamentally unstructured. Anyone can add a callback, or await a promise at any time, and once that task is created, it has an unconstrained lifetime. If the async function or the callback is no longer relevant, how do you cancel it? Worse yet, if you're performing multiple callbacks or promises or awaits, are you sure you're dealing correctly with errors in all of them?

Introducing Effection

The solution is to adopt the ideas of structured concurrency and apply them to JavaScript. The core idea of structured concurrency is that the lifetime of a task is always constrained. This means that a task must not outlive its parent, or in other words, it is impossible to create a task that runs forever. This might seem like an obvious constraint, but it is important to note that this is very much not the case with the core concurrency primitives available in JavaScript.

Effection is an Open Source concurrency framework that replaces async/await with a more structured way of writing code. Let's look at how this applies to async/await code:

async function fetchUser(id) {
  let response = await fetch(`/users/${id}`);
  return await response.json();
}

There is nothing special about this async function. And this is what happens when we call this function from another function:

async function fetchSomeUsers() {
  let userOne = fetchUser(1);
  let userTwo = fetchUser(2);
  return {
    one: await userOne,
    two: await userTwo,
  }
}

Here we're calling fetchUser twice, and then awaiting both calls. This will cause both calls to fetchUser to execute concurrently. We could also implement this using Promise.all, which would behave similarly and is subject to the exact same pitfalls:

async function fetchSomeUsers() {
  let userOne = fetchUser(1);
  let userTwo = fetchUser(2);
  return Promise.all([userOne, userTwo]);
}

It looks like the fetchSomeUsers function is nice and self-contained, but in fact it isn't. We start fetching two users, but both of those fetches are in no way tied to the fetchSomeUsers function. They run in the background, and no matter what happens within fetchSomeUsers, they just keep running. Potentially they could run forever; that's what we mean when we say that their lifetime is unconstrained.

For example, there is nothing stopping us from doing something silly like this:

async function fetchUser(id) {
  setTimeout(() => {
    console.log("I'm still running, lol!");
  }, 1000000);
  let response = await fetch(`/users/${id}`);
  return await response.json();
}

Even ages after the fetchUser function has finished, it will still print something to the console. We can't close the existing loopholes (such as setTimeout), but as long as you are writing idiomatic Effection code, something like the above just cannot happen.

Let's look at this example again – but this time using Effection:

import { spawn, fetch } from 'effection';

function* fetchUser(id) {
  let response = yield fetch(`/users/${id}`);
  return yield response.json();
}

function* fetchSomeUsers() {
  let userOne = yield spawn(fetchUser(1));
  let userTwo = yield spawn(fetchUser(2));
  return {
    one: yield userOne,
    two: yield userTwo,
  }
}

It's not that different from before: Effection uses generator functions instead of async functions, and there is something going on with spawn, but other than that it looks pretty much the same.

However, the way that this function runs is very different. When we spawn a task with Effection, its lifetime is tied to the current task, which means that fetchUser(1) and fetchUser(2) cannot ever outlive fetchSomeUsers. Moreover, no task that fetchUser spawns can outlive fetchUser either. Effection tasks ensure that everything that happens within the task stays within the task.

Using Effection

This might seem like a trivial outcome but the implications are profound. For example, Effection ships with an operation called withTimeout, which adds a time limit to any task. If the time limit is exceeded, an error is thrown.

That means we can now do this:

import { withTimeout } from 'effection';

function* fetchSomeUsersWithTimeout() {
  return yield withTimeout(2000, fetchSomeUsers());
}

Right now in JavaScript something like this is pretty much impossible to implement with promises. There is just no way to know what fetchSomeUsers does internally, and there is no way to know whether we really can abort all of it. Trying to write a function like this is at best unsafe and at worst impossible.

With Effection it all just works. We know that nothing can escape fetchSomeUsers and we know that everything that fetchSomeUsers does will be canceled if we cancel fetchSomeUsers itself.

Now, imagine that our async/await fetchSomeUsers fails for some reason:

async function fetchSomeUsers() {
  let userOne = fetchUser(1);
  let userTwo = fetchUser(2);
  throw new Error('boom');
  return {
    one: await userOne,
    two: await userTwo,
  }
}

What will happen in this case? fetchUser(1) and fetchUser(2) will happily keep running, even though the fetchSomeUsers function which initially called them has already failed.

fetchSomeUsers timing with async/await

This can't happen in Effection. Because given that fetchUser(1) and fetchUser(2) are scoped to their parent function, they will be terminated when fetchSomeUsers enters into an error state.

fetchSomeUsers timing with effection

And this is the power of Effection's structured concurrency: it allows us to build abstractions that would otherwise be impossible to construct. We think it is a fundamentally better way to write JavaScript.

Going further

There is much more to Effection than what we've shown here. Effection is not only a framework that allows you to write rock-solid code, but it also gives you amazing insights into your application through our experimental inspector. If you're curious to learn more, this guide explains in greater detail how to use Effection, and the API Reference contains a complete reference of all methods and types that Effection provides.

We are a small but dedicated team at Frontside that likes to tackle challenging problems with an ambitious solution. There is boundless work to be done, and boundless ideas to explore. Do you want to join us on this journey? Come hang out with us on discord, where we stream our work and are always interested in discussing where to go next.

Interactors: the design systems testing ally

Charles Lowell, Jeffrey Cherewaty — Wed, 04 Aug 2021 05:00:00 GMT

The reusable component libraries shipped with design systems enable developers to use on-brand and battle-tested components. Developers don't have to re-invent the wheel every time they need a common element like an input, modal dialog, or date picker.

Screenshots of date picker components highlighting clickable areas

If you're building an application with some of these components, you'll want to write some tests. With the date picker component pictured above, a user needs at least four clicks to select a date. That means a UI test will also need to step through those four clicks. Testing this interaction with React Testing Library and Cypress could look like:

cy.querySelector('#date .MuiCalendarPicker-root').within(() => {
  cy.getByLabelText('calendar view is open, switch to year view').click();
  cy.getByText('1992').click();
  cy.getByLabelText('Next month').click();
  cy.getByText('16').click();
});

To effectively write a test using this date picker, we had to include specifics about its implementation, like IDs and classes. If the underlying component changes its DOM structure, our tests are suddenly brittle.

There's a well-trod design pattern for fixing this problem: page objects. They're object-oriented classes that serve as an interface. As an application matures alongside its accompanying tests, page objects act as a buffer and a more explicit API for the tests to interact with the application.

Introducing Interactors

At Frontside, we've built Interactors (@interactors/html), a library inspired by page objects that helps teams structure, share, and reuse their UI testing practices.

Interactors evolve the idea of a page object. Modern applications are usually arranged into composable components, and the "page" is no longer the dominant unit of organization. An Interactor is similarly composable, and can abstract any level of object in the DOM hierarchy.

Design systems maintainers can build interactors alongside components to reduce the mental distance between the user interface and its testing-friendly abstraction. In the same way strong typing can make components easier to build with, interactors make components easier to test.

Interactors...

... are composable, so they make tests easier to write, read, and run.
... act as an abstraction layer on top of the component's implementation, improving design systems' maintainability.
... have TypeScript support and provide helpful errors, resulting in an improved developer experience.

Let's take a closer look at each of these advantages.

Interactors make tests easier to write

We provide basic built-in Interactors that correspond to HTML elements, like button, link, and checkbox. Using these alone, you could interact with and assert the majority of cases of a web UI. Take the following example of a collapsable navigation menu test:

import { Button, Link, Heading } from `@interactors/html`;

it('goes to international news page with mobile menu', () => {
  cy.do([
		Button({'aria-label': 'Show Navigation Menu'}).click(),
		Link('International', { 'class' : contains('nav-link')).click()
	])
	cy.expect([
		Heading('News').exists();
	])
});

You'll often want to avoid repeating yourself and pack together interactions and assertions that correspond to your design system's components.

It's likely that Nav is indeed a component in our design system. Thus, we can create a Nav Interactor that queries the nav and has actions that a user can perform with it.

it('goes to international news page with mobile menu', async () => {
  cy.do([Nav().goTo('News')]);
});

it('goes to entertainment news page with mobile menu', async () => {
  cy.do([Nav().goTo('Entertainment')]);
});

Note that we can target the Nav directly and perform the action we care about in a single line. If the component changes its internal classes or markup, these tests won't break. Its Interactor received updates to account for those changes.

Retaking the example from the introduction, you could provide a DatePicker Interactor such that everyone using the pick date component have standard methods to test their features using an interface like:

cy.do(DatePicker.setDate({ day: 16, month: 'August', year: 1992 }));

Composable Interactors also respect their lifecycle in the browser. That means that they don't waste time waiting on dead components or looking for new ones when there's no rendering happening.

Interactors improve maintainability in a design system

Interactors rapidly become design system's maintainers' best friend because they provide freedom by abstracting the testing practice away from the component's implementation. Instead of relying on fragile internal component classes, developers and maintainers can use Interactors as an API contract for the UI.

Typically, developers would target elements in the UI in their tests by reaching into the internals of the components they use. It's common to see tests that target chains of selectors like this one:

#notes-modal-notes-list [class*="mclRowFormatterContainer--"]

However, referencing internal selectors couples the test implementation to the HTML structure of the component; a slight change to the markup may cause tests to fail. Thus, tests are fragile and introduce fear in the design system's maintainers because they have no way of knowing what might cause tests to fail. Worst of all, this fragility makes tests unreliable when updating the design system, which is precisely the kind of system-wide change that you write your tests for.

Interactors are the missing contract between the design system's maintainer and its users. The maintainers can control the API that test authors use to interact with the component. The maintainers can freely change the markup of components without worrying about inadvertently breaking tests. And test authors have a more convenient way to write their tests. In the end, the tests are more reliable and provide more confidence to everyone.

Interactors improve developers' experience while testing

Interactors know more about the components under test than mere selectors, which allows them to provide helpful information while you develop or debug tests. Having an explicit API to test your app provides static checking safety, and understanding that API enables Interactors to provide suggestions when things go wrong in a test.

When you write your test assertions and actions using selectors, you're on your own. There are no checks available while you write the test, or while it compiles. You'll only find out about a typo after running the tests, and inspecting closely why your test is failing although it seems that it should pass.

Interactors bring in static checks for your tests, which means your IDE and compiler can provide more support as you develop:

Screenshot of IDE showing in-line suggestions and documentation about Interactors

Not everything can be found by static analysis in a test, but Interactors are ready to provide helpful suggestions for common small mistakes that are usually hard to debug. For example, if you were looking for a login button in your test case, but forgot for a moment that the button actually reads "Log In", Interactors will hint that to you:

Screenshot of interactors suggestion similar elements to the one that was not found

Interactors not only check for presence before committing an action in the UI, they also check if the element is visible—through various heuristics—and that it is enabled. For example, if Interactors did find the button and it is visible but not enabled, it will throw an error like this:

Screenshot of interactors failing a test because the target element was visible but not enabled

Try out Interactors!

If you're still not sure about trying out Interactors, take a look at this pull request in FOLIO, an open-source project, adopting Interactors in their component library:

Screenshot of code diff resulting in refactoring a test using React Testing Library to use Interactors

The selectors are difficult to follow and are quite fragile, while the Interactors are easier to read and focus on testing the app as a user would use it.

You can start using Interactors as part of your current test setup, they're compatible out of the box with Cypress and Jest, so it's easy to get started!.

There and Back Again: Testing and Stack Upgrades

Upgrading your application stack to new major versions can be terrifying, especially when you have a massive application with 75 developers working on features non-stop. However, a good testing strategy gives you confidence to make bold moves that take your organization to heretofore unexplored lands.

Continue reading →

Unleashing the Genie: Helping both designers and developers with Storybook

Good documentation is key to design systems. However, what "good" means varies according to the reader. This article will go through tips to generate documentations for designers and developers from a single Storybook—keep everyone happy with minimum effort!

Continue reading →

Relieve developers’ churn in your Cloud native strategy with Backstage

Taras Mankovski — Fri, 14 May 2021 05:00:00 GMT

When you start breaking your monolithic platform into microservices and deploying components independently, you can’t help but feel excited about adopting a Cloud native strategy. You see releases becoming more frequent and smooth and your teams feel more productive and motivated as they see their work rapidly move into production.

It’s all great – that is until you hit a critical fragmentation point. As you keep adding microservices to your Cloud native platform, it gets harder and harder to keep track of what’s going on. Who’s building this service and with which stack and patterns? Why do we have three authentication mechanisms? Where are all these Cloud provider bills coming from?

The landscape for your developers is not much better. They must launch into detective mode and trace APIs and codebases in order to figure out the services they need to build their features. Then they need to figure out how to deploy their own work, which usually turns into an epic quest against the Cloud monster.

The problems with Cloud native practices seem to proliferate with each passing day, yet companies keep pushing towards their adoption across their teams because of its proven track record of increasing software development velocity. What’s a developer to do?

Thankfully, soon everyone will have access to Backstage: an all-purpose platform to make the Cloud native experience better for everyone!

Backstage is an Open Source platform for creating Developer Portals. Currently a Cloud Native Computing Foundation sandbox project first developed by Spotify, Backstage is getting a lot of traction among organizations experiencing the problems of microservices at scale when combined with the challenge of supporting dozens of developers on their team.

In this article we’ll cover the challenges developers experience using Cloud native strategies and how Backstage alleviates them. We’ll use the example of a secrets manager to illustrate just how useful Backstage is. By the end of the article I guarantee you’ll want to slip Backstage too and see what’s got everyone excited!

Churn ‘n Burn

Organizations who adopt Cloud native strategies do so because they need to in order to stay competitive. Users cannot bear the traditional cadence where a special commission would sit down for months to release one feature. Your customers expect you to deliver new features faster and improve the user experience all the time, and will switch to the competition in the blink of an eye if they’re not getting that from you. Having no bugs is the baseline now, so whenever one sneaks into production you need to be able to roll back immediately. Kubernetes provides you with the automated technology to make all of this possible. But it's an open secret that Kubernetes is quite difficult to use, and that has a direct effect on how your teams operate. What’s gotten lost in the transition to developing Cloud native apps is an awareness of your team’s experience and their comfort level with the tools they use.

Since the inception of Kubernetes in 2014 and throughout its evolution, a whole industry has spawned to help companies adopt it. And no wonder: Kubernetes is not an easy technology to learn and manage due to the enormous complexity that it’s designed to deal with. For that reason, most Cloud providers have created managed-Kubernetes platforms: Microsoft Azure AKS, Amazon EKS, Google GKE, and IBM CloudFundry are just a few of the most obvious examples. Each of them imagined a UI that would work nicely to manage Kubernetes, and countless other vendors offer tools to monitor and automate different aspects of Kubernetes, each with their own interfaces and abstractions as well.

While the attempt to support the adoption of Cloud based practices is to be applauded, what’s emerged is a very complicated landscape for those people tasked with implementing Kubernetes. Not only must they master this intricate technology, but they must also be proficient at the diverging set of tools that serve it, which change and evolve as fast as Kubernetes itself. Oftentimes, engineers wind up having to manage two or three Cloud solutions providers at a time, forced to use Microsoft’s Azure because of a company-wide policy but also required to handle Amazon’s AWS to spin up macOS containers on demand.

Nor is it just your DevOps team who are experiencing the churn of Cloud native solutions. Developers are also taking the hit with today’s trend of self-service deployment setups. Developers are not only expected to write robust apps, understand the business, and coordinate with other teams – they’re asked to be Cloud native too, which implies getting to know an entirely new world of complexity that seemingly changes every six weeks. That much churn puts developers at risk of burnout.

A one-stop tool shop

Backstage’s mission is to empower developers. Instead of creating yet another platform to manage the infrastructure that supports Cloud native solutions, Backstage’s innovation is to create a portal to manage the organization’s services. While the infrastructure is a critical piece of any service, Backstage sees it as a means to the real end: achieving business goals.

Backstage brings together in one place the various aspects of each service: its codebase, documentation, environments, infrastructure, and of course the people who build them. The teams like Spotify who are adopting Backstage are learning that investing in their developers’ experience pays off. When working with Frontside’s enterprise clients, I see the true magic of Backstage is in how it provides developers real ownership of what they’re building. It’s truly giving them the keys to the castle.

Yet a funny thing happens when you give someone the freedom of ownership: it transforms into responsibility. When the people who design and develop your software have the freedom of ownership over their services or apps, their creativity and productivity goes through the roof. But with that freedom comes a genuine sense of responsibility to deliver best-in-class experiences.

Prior to Backstage, teams would often be “ready to ship,” only to discover that being merely feature-complete isn’t enough – they still had an unknown path to go down before their work reached production stage. For a developer, having to set up a ‘war room’ to push the update or app through the development path is the equivalent of trying to get spaghetti to roll uphill – a frustrating and often fruitless experience.

But with Backstage the development experience is transformed. For example, by using Backstage Scaffolder, developers can easily set up work environments that resemble production more faithfully, resulting in less friction and fewer surprises along the development path. They can start developing apps that are compliance-ready and infosec-approved from the very beginning by pulling in pre-made solutions and proven architecture patterns that are shared in Scaffolder.

Thanks to Backstage’s Service Catalog, teams can break through heaps of emails, meetings and bureaucracy by bringing together all the aspects of every service under a single portal. With a 360-degree view of the development process, your team will be able to discover new synergies, generating opportunities for joint efforts while avoiding duplicated solutions across the organization. Now that all your services are visible in a single portal, you can also find orphaned projects and take care of them.

Don’t tell a soul: Secrets management

Every organization’s Cloud platform needs a way to manage secrets and apply them to containers. The easiest place to start is to use Kubernetes' built-in secrets management. This approach is easy to set up but difficult to manage and secure long term. A more permanent solution is to manage secrets via a Git repository with GitOps workflow or by using a secrets vault like Azure Key Vault or Hashicorp Vault. Some organizations adopt a more-the-merrier approach and use all three of these in sequence. The options that are available to your platform team might depend on procurement practices of your organization or its familiarity with these tools.

But when it comes to secret management, it’s not just the UI that changes across providers: each platform has their own idiosyncrasies. For example, the requirements in minimum length for secret values are different in Microsoft Azure and Hashicorp’s Vault. In short, there’s a lot of variables to keep track of. At least one thing is certain, however: expecting your developers to fluidly switch between three different ways of managing secrets will make them want to tear their hair out.

Screenshots from different secret management UIs - they all look different, yet do the same thing!

Screenshots from four secrets management UIs from different providers

A better developer experience would provide your developers with a consistent and familiar interface for managing secrets no matter which Cloud providers come and go. That’s Backstage in a nutshell.

Given that it already aggregates services and other assets, a natural place for such an interface to live is in Backstage. And the benefit of building it in Backstage is that instead of having your developers deal with specificities in a UI, you can abstract them away in a unified API for your aggregator microservice. Your secret manager UI in Backstage will reflect these unified rules, removing these complexities from your team.

Screenshot of a possible secret manager implemented in Backstage

A Prototype of a Secret Manager plugin build on Backstage

And best of all, your team already knows how to build a Backstage plugin like this! Somebody familiar with React and the company’s microservices setup can take up the task and have it done in no time at all.

Architecture diagram for the plugin. The secret manager plugin is inserted in Backstage's UI, which is all rendered in the user's browser. This UI communicates with an aggregating microservice, that contacts external Cloud providres

Basic architecture diagram for the Secret Manager plugin

In Backstage, plugins are React components that get added to each service’s page. Thus, the secret manager plugin will be a React component with a UI where your developers will create and edit secrets. This component will make HTTP requests to a microservice that will be responsible for retrieving and editing secrets in your current Cloud providers. Going forward, your team will only have to adjust the aggregating microservice whenever there is a change in Cloud providers.

Conclusion

Backstage not only allows you to keep evolving your Cloud native strategy without compromising the productivity of your developers: it empowers them to take your organization to the next level by uncovering collaboration possibilities and promoting shared solutions on top of the Cloud. Secrets management is but one example! The possibilities are quite broad for expanding Backstage to fit in your needs, from Enterprise API integrations with tools like WSO2 or Mulesoft, to putting together infrastructure workflows for data processing.

Unleashing the Genie: Helping both designers and developers with Storybook

Varya Stepanova — Wed, 07 Apr 2021 05:00:00 GMT

The goal of a design system is to have everyone using it across apps, products, and teams, but achieving that is easier said than done. It’s no mystery that the easier a design system is to use the more uptake it will get. But a design system’s ease of use is oftentimes contingent on how good its documentation is.

Using Storybook can move your team a long way towards creating effective and easy to use documentation for a design system. Storybook allows you to showcase “stories” for each of your design system’s elements, including permutations of their visual and functional options. As your design system grows, you can add to the stories you tell about it.

However, teams often hit a point in which there’s so much information that Storybook becomes hard to use and navigate. Designers have to scroll through APIs that are not relevant to them, or decipher technical names that have been generated for their UIs. Developers similarly have to skip through the various design edge-cases to find the parameters they’re interested in. Too often what was once a manageable collection of stories becomes as long as the classic One Thousand and One Nights.

One obvious solution to arrive at more usable documentation for both groups would be to create a version for designers and another one for developers. But maintaining two Storybooks would require a lot of work and result in a fair bit of duplicated effort.

As it turns out, we don’t have to keep two separate Storybooks to achieve the same usability goals for both designers and developers. In this article, I’ll explain ideas to generate Storybook stories straight from your codebase, and thus avoid maintenance churn. After that, I’ll overview techniques to generate two different sets of documentation from a single Storybook using build options.

Documenting Design Tokens

Storybook allows you to document your stories with @storybook/addon-docs, including design tokens and other elements of your design system.

For example, you can use blocks from @storybook/addon-docs to document your color palette:

import { ColorPalette, ColorItem } from '@storybook/addon-docs/blocks';

export const Palette = () => (
  <ColorPalette>
    <ColorItem title="Primary" subtitle="Pink" colors="#F74D7B" />
    <ColorItem title="Secondary" subtitle="Dark" colors="#15325D" />
    <ColorItem title="Tertiary" subtitle="Blue" colors="#26ABE7" />
  </ColorPalette>
)

Similarly, there's other blocks available for documenting other design tokens such as typography (<Typeset>) and iconography (<IconGallery> and <IconItem>).

The full documentation for these blocks is yet to be completed, but in the meantime you can read an explanation about them in detail in Storybook’s team announcement: “Rich docs with Storybook MDX." While inputting design tokens documentation by hand is simple, it can be redundant because we have already stored those values in our codebase. We can leverage that existing code to guarantee up to date documentation – and avoid extra work.

In the case of the color palette, we don't need to list the colors manually: as we usually store them as a JavaScript hash, we can iterate and display them all one by one as shown in this example:

import { ColorPalette, ColorItem } from '@storybook/addon-docs/blocks';

import { colors } from 'path/to/tokens.js';

export const Palette = () => {

  return (
    <ColorPalette>
      {colors.map(color => <ColorItem title={color} subtitle="" colors={[color.value]} />)}
    </ColorPalette>
  )
}

We can extend this idea to document helper classes that assign specific text colors, background colors, margins, padding, and borders. These kinds of classes are typically used for fast prototyping. Helper classes are handy for making quick changes and variations like these:

Screenshot of texts with classes applied on a story

Classes are not stored in a hash, but we can iterate through CSS classes directly just as we did for the colors.

You know the saying: it’s easier to show than tell. So instead of just listing the available classes, you can document them applied to a <p> element so people can immediately see the effect of the class. An implementation could look like this:

import styles from 'path/to/_helper-classes.scss';

export const textColors = () => {

  const textColors = Object.keys(styles).reduce((acc, key) => {
    if (key.startsWith('color--')) {
      acc.push(key)
    }
    return acc;
  }, []);

  const textColorSamples = textColors.map(cl => (
    <div className={styles[cl]}>Text color with .{styles[cl]}</div>
  ));

  return (
    <div>
      {textColorSamples}
    </div>
  );

};

As you can see, the story imports the SCSS file used in the codebase, which means even the documentation of helper classes is automatically up-to-date and accurate.

We can generate stories from our codebase for other types of design tokens. For example, if your design system contemplates button variations or icon options, you can take them from your code and display them automatically as well.

But here’s where things can get messy. Let’s take a closer look at the example of icons. A single project can use hundreds of icons, and each can be available in light, regular, or filled – not to mention that each of those can be available in different sizes. If we generated a story that displayed every possible permutation of the icons available from the code, we’d get a grid with thousands of icons that would endlessly go on like Scheherazade speaking to the king. Nobody wants to scroll infinitely trying to find a single icon!

When we’re confronted with such a situation, bringing in some interactivity to our story can help. To make the list of icons more useful, you can implement a component that generates all of the icons from the codebase but offers filters to show only those that are interesting for the user, depending on type, size, and even name:

Screenshot of story with generated icons

As you can see in the screenshot above, generating the icons from the codebase has another feature (or challenge): the names are written as they are in the codebase, like <IconEmailFilled />. For developers, this is great because that’s exactly what they want to know. However, designers may find such names quite confusing, especially if your designers are consuming your stories through a tool like ZeroHeight. Is there a way we can summon Aladdin’s genie from the lamp and have him grant us a wish to solve this conundrum?

Different users, different stories

Designers and developers are interested in different aspects of a design system: a designer wants to know about edge-cases in the UI like a button with a very long text, while developers want to know about how onClick works in a given component. Trying to cobble together both perspectives in the same story doesn't work very well, so instead I usually write separate stories for developers and designers. Before you raise your hand, let me explain that I don't maintain two Storybooks (that'd be hard): I host two different kinds of Stories in the same Storybook but in order to avoid the copy-paste one is usually inherited from another. They are also seasoned with different decorators, bringing only the desired experience to their target audience.

My Storybook is implemented so that the same component has some stories that describe it from the design perspective and some that are good to follow for developers. Mostly, the difference is in what I explain about each component's story. In the design-oriented stories, I show what visible variations a component might have and the design edge cases. While in the stories for developers I focus on functional features of a component.

However, not every component needs separate stories for designers and developers. There's cases when you only need to make a small tweak to the content to make it fit the needs of developers or designers.

Let’s go back to that list of all icons where the names were equal to the corresponding React components and therefore served the needs of developers. For designers, I coded a second story. Instead of showing the react component name, it displayed the icon’s name from the component’s meta data. Since I coded the icons story to be generic enough, I didn't have to make any copy-paste to create two stories. Instead, I only need to reuse the story and tell it which kind of name to use by passing in a parameter. The designers’ story would look like this:

Screenshot of story with generated icons with designer names

It is possible to keep pushing towards a more refined experience for our readers depending on how they consume Storybook. For example, in a project I lead, developers used a special UI to keep documentation on sight on a side screen while they coded. In that case, a different background and a thicker stroke around stories came a long way to provide a nicer UX for them. However, those styles would look odd for designers who only consumed Storybook through a regular web view.

You can generate two documentations with different styles without having to manage two Storybooks or make any copy-pastes in the process. The key is to rely on decorators to apply special styling (and do any other needed processing to the story) depending on its target audience;

For instance, you can implement a @developerStory decorator that would only be applied to the stories kept in the files with a special .docs.stories.js.This decorator performs all the necessary style adjustments to the display of the stories when they are built. And you can do this all with a single line of code! It looks like Alladin’s genie (or rather Storybook’s genie) came through for us.

When it’s time to ship our documentation to their target users, having enriched stories allows you to have more granular control over how the production build will look like. You can go as far as having two different Storybook configurations that you toggle with a flag on the build command. Or you can decide to not include stories for developers for a target site (for instance if it'll be consumed by a tool like Zeroheight). The opposite outcome is also possible: you can build a site that targets developers only and not show the design-centric stories.

In other words, pretty much anything you could wish for.

Conclusion

The tips and tricks written above are indeed not the only things that one can do to improve the Storybook setup. But they nicely illustrate how you can tailor your team’s Storybook to improve your design system's adoption across your organization and make its users more satisfied.

Varya Stepanova is a design systems architect—and co-creator of BEM—based in Finland. Varya has improved the way dozens of teams, large and small, develop and evolve their design process towards better collaboration among stakeholders.

There and Back Again: Testing and Stack Upgrades

Taras Mankovski — Thu, 18 Feb 2021 05:00:00 GMT

Upgrading your application stack to new major versions can be terrifying, especially when you have a massive application with 75 developers working on features non-stop. The situation gets more complicated when you also have important deadlines pressing down on you, new defects popping up every day, mounting tech debt, and 30% of the team is new to the company. What should be prioritized and what can wait? My call in 2018 was none of the above. Instead I reached for J. R. R. Tolkein for inspiration to find the one “ring” that would address all these issues in one fell swoop. After surveying the options I put automated testing at the front of the queue. But unlike in Lord of the Rings, automated testing didn’t cause more problems than it solved – in fact my decision paid off big time.

I was hired by a major bank who had recently acquired a development firm in India and was facing tough challenges within their engineering organization. They had brought in several consulting firms to do staff augmentation, and everyone was doing their best to keep hitting delivery targets. But every step forward seemed to be met with two steps backward, and the quality of the product was in decline. They started to look a lot like Frodo and Sam going in circles on their way to Mordor.

It turned out that part of the problem was that the application was being built on top of an early and balkier version of modern JavaScript. A newer version of the framework allowed lazy loading parts of the application, which could solve performance issues that they were experiencing. Upgrading was both necessary and urgent so the team could use open source solutions that were available without requiring them to completely rewrite the application.

But how do you go about upgrading an application that is under active development? First of all, you need a way to tell that everything works properly throughout the upgrade process. There were more than 10 rounds of upgrades to perform to take the app to the most recent one, and each one might prove to introduce complications rivaling those that Saruman put in the way of the Fellowship of the Ring. Frodo and Sam went on foot to Mount Doom, but as anyone familiar with the epic knows their journey was interminably long. Manual tests would face the same fate, making the upgrade process impossibly slow, especially in light of the size of the app in question.

What we needed was a test suite and a robust one at that. The codebase had a few unit tests sprinkled here and there but they didn’t inspire any more confidence than the “assistance” of Merry and Pippin did. QA had a test suite but it was too slow and unreliable to run on every commit. We needed one that developers could use locally, that would run quickly on CI, and that would give us the confidence that we didn’t break anything. Our tests had to put the whole app under the microscope and ensure everything worked correctly from the users' perspective. This is what I call “testing big.”

Building that test suite was the problem that we had to address first. It took a good deal of convincing to get their technical leadership on board with prioritizing testing in the midst of so much pressure. When I first floated my proposal to switch gears and prepare tests in order to upgrade the application stack, the architects in the room looked at me as if I was as crazy as Frodo in setting out to destroy the ring.

Yet I was confident that robust testing and a sound tech stack was the most urgent need for the company to address. Developing a test suite is relatively predictable, and once you have tests that you can trust, even scary upgrades become predictable. My confidence in testing did not come from a false belief in my own ability but like Frodo from a belief in those around me. In this case it stemmed from the experience of the entire Ember.js community and ecosystem, which had made testing a fundamental part of their mindset. The framework the client was using was Ember 1.13 and we were aiming for 2.8 – an impossible task in the eyes of many at the company. But what they were not aware of was the framework’s hidden strength: seamless upgrading thanks to automated testing. The combination of the two is very effective, and community surveys show that 41% of the users use the most recent version, and more 80% use the latest major version.

I was on site in Bangalore leading the team who would be responsible for writing this first test suite. The client assigned six engineers to the newly formed team – those who had the most knowledge about the app. We had six weeks to succeed or die trying.

We first developed a test harness, which required creating simulation mechanisms for authentication and the backend responses. The process was a great learning experience, as we discovered patterns in the app and found implementation problems in several sections. By the end of our time together we had finished the test harness, refactored the application to use tests, and had guidelines for creating new tests. After a celebratory dinner of masalas I’ll never forget (no lembas bread for us) I returned home to Canada. In the meantime, the team managed to merge all the new features that had been put on hold while they were writing tests. Better yet was the fact that they were able to use the test suite to make sure that the new features didn't break anything in the app – a wholly new experience for the team. They additionally had developed strict testing policies to accept new PRs.

Once the codebase was fully tested and stable, I returned to Bangalore to drive the upgrade process. The process was predictable as we were following deprecation warnings progressively, and fixing failing tests as we moved along. The team discovered many unpleasant surprises in the codebase as we refactored through the app. Just as Frodo grew in his knowledge as a result of his journey, I used those cases as learning opportunities to explain to the team why change in their development process was necessary. But we were able to go through two years of evolution of the framework smoothly in just six weeks.

Not only did we do a massive upgrade, we had tested the entire application and developed a testing mindset across the frontend team. And this was all done by a small team who had no prior experience testing or upgrading tech stacks.

Long story short: investing in your test suite pays off, but you need to make sure that that investment is covering your business value: your users. Testing big gives you confidence to keep pushing forward, knowing that you can try new things because your app functionality is protected. Without the fear of costly mistakes your creativity increases, and like Frodo you have more time to make bold moves that take your organization to heretofore unexplored lands.

Kubernetes: Dip your toes in this

Min Kim — Fri, 29 Jan 2021 05:00:00 GMT

Kubernetes is perhaps one of the most exciting developments in the DevOps world in the last 15 years. It is an open source implementation of the Borg system that was Google’s “secret sauce” – the thing that allowed them to deploy and manage thousands of services in their clusters.

With modern cloud PaaS (platform as a service) like Google Cloud and AWS, most frontend and many backend developers can deploy services without learning about the architecture and technologies that power these platforms. In the past, not knowing about these platforms could be easily forgiven because the learning curve was too steep.

Kubernetes changes all that. It makes understanding these platforms approachable and manageable, allowing frontend developers to learn all about them fairly quickly. The architecture of Kubernetes and the approach that the community is taking towards building Kubernetes continuously lowers the barrier for people who otherwise wouldn’t have an opportunity to learn about Cloud Native Deployment.

At its core, Kubernetes is based on a concept that anyone familiar with React would recognize. It’s a declarative system for service management. Similar to how React allows you to declare what components you want to render with it handling the creation of DOM elements, Kubernetes allows you to declare what services you want to deploy and it takes care of deploying them.

Grasping this comparison can take you surprisingly far into understanding Kubernetes. For example, you can use the same components’ syntax to describe their composition for both web and mobile platforms. React DOM and React Native are adapters that control how your components are rendered in these different environments. Similarly, you can use the same services and same syntax to deploy services to Google Cloud, AWS, Azure or Digital Ocean via different cloud providers.

React takes care of the difficult work of applying declarative state changes to DOM. Kubernetes does the hard work of applying declarative state changes to your infrastructure and services. The important part to remember is that with Kubernetes, you tell it what to do rather than how to do it.

Under the hood, Kubernetes consists of a control panel that manages worker nodes. The control panel’s API is RESTful, but most of the time you’re not going to be interacting with the API directly. Instead you’ll use the CLI, which talks to the RESTful API. The control panel’s specs are agreed on by the Kubernetes community, but the API definition makes it possible for different implementations of Kubernetes to exist.

In this tutorial, we’ll show you how to set up MicroK8S, which is Ubuntu’s implementation of Kubernetes. It’s small enough to run on your machine and will allow you to play around with Kubernetes without having to pay a cloud provider to host it for you.

Installation and Start

Let’s go ahead and install MicroK8S for macOS:

$ brew install ubuntu/microk8s/microk8s

For installation instructions for Windows or Linux, go to the MicroK8S website.

Once it has finished installing, run the following commands:

$ microk8s install
$ microk8s start

microk8s install will start the VM and microk8s start will create a Kubernetes cluster.

Once you start up a cluster, you should be able to run the microk8s status command to display all of the available services. Entering microk8s stop into your terminal will shut off both the VM and the cluster.

Dashboard

Although all of MicroK8S’ functionality can be accessed via your terminal, you can also start up the dashboard service and see everything through the web UI.

Run microk8s dashboard-proxy and you should see the following output:

$ microk8s dashboard-proxyChecking if Dashboard is running.
Waiting for Dashboard to come up.
Dashboard will be available at https://xxx.xxx.xx.x:xxxxx
Use the following token to login:
abcdefgh1234567
Forwarding from 0.0.0.0:xxxx -> xxxx

Go to your browser and enter in the address provided (including the https://) and copy/paste the token to gain access to your dashboard, which will look something like this:

Microk8s Dashboard screenshot

Now when you go back to your terminal, entering the microk8s status command will show that the dashboard has been enabled. If you wish you can run microk8s disable dashboard to disable it again.

Kubectl

Coo-beck-tul? Cube-control? However it’s pronounced, kubectl is a MicroK8S command to help you control Kubernetes clusters; like the ones we started earlier with microk8s start. Let’s use kubectl to deploy a service. For this tutorial we’ll be using a lightweight microservice: https://github.com/stefanprodan/podinfo. The service itself does not do much. It is just to help us practice starting and stopping Kubernetes services.

Use the following command to deploy Podinfo:

$ microk8s kubectl apply -k github.com/stefanprodan/podinfo//kustomize

Now you can run microk8s kubectl get deployments to confirm that podinfo has been deployed.

k9s

We haven’t port-forwarded our deployment so we can’t access it with a browser just yet. There’s a way to port-forward with microk8s but we’ll show you an easier way using k9s.

You’ll need to first install it:

$ brew install k9s

After it finishes installing, you can start it up using the command k9s. Your terminal should look something similar to this:

k9s running on terminal screenshot

You can exit out of k9s with either ctrl + c or you can press : to initiate a command and enter quit.

In order for k9s to work properly, you’ll need to add your Kubernetes configuration to your local root directory:

$ mkdir ~/.kube && microk8s config > ~/.kube/config

And now when you start up k9s again, you’ll notice the configurations have taken effect.

If you’ve been following along with all of the previous steps, your k9s screen should show the pods that we deployed earlier:

k9s with pods listed screenshot

What we’re seeing here are the pods from the service we deployed. We’ll be returning to this page very shortly, but for practice if you want to navigate to the list of deployments, press : and type in deployments.

k9s with deployments listed screenshot

To port-forward this deployment, press enter on podinfo to display its pods again.

Navigate to any one of the pods and press shift + f to port-forward the service. Then press f to see the hosted URL. Copy and paste this URL to your browser and you should be able to see the service.

Podinfo screenshot

Conclusion

Kubernetes is on the rise and having it in your toolkit can get you a long way. Now that you’ve set up a MicroK8S in your machine, you can play around with the basics to start getting familiar with Kubernetes.

The context dilemma: design tokens and components

Varya Stepanova — Fri, 15 Jan 2021 05:00:00 GMT

Design systems are growing in popularity as organizations realize how greatly they empower their various teams to create accessible on-brand web and mobile apps while avoiding duplicated work. However, they often encounter conflicts when mapping established style architecture patterns onto the new abstractions inherent in a design system. In this article I want to talk about a common problem: context and naming around shared variables.

On one hand there are design tokens, which have been used successfully by countless teams over the years to ensure that elements across their UIs are consistent. There are design tokens for fonts, colors, sizes, spaces, and even indivisible pieces of an interface. Designers want every part of the app interface and all the elements to have the same look and feel, and design tokens were created to help them achieve that outcome.

On the other hand, design systems emerged to facilitate easier development with premade components while ensuring that style remains consistent. The team employs them to reuse elements and functionality across the app. Components include branded date-pickers, buttons, modals, and other functional elements of an interface. They too can be used across different sections of an app, but their implementation is oriented locally: the styles and logic are written as if the component is an independent element.

Reuniting these two worlds can be conflicting: how do you name design tokens semantically if they have to make sense everywhere but also respond to the inner logic of individual components which are independently developed? In this article, we'll explore the problem and how BEM—a well-known naming design pattern—can come to the rescue.

The crux of the matter illustrated

Let's say we are defining design tokens for colors. We give them semantic names to reflect the meaning of each color:

$dt-primary: #f74d7b;
$dt-disabled:  #767677;
$dt-inverse: #FFFFFF;
$dt-lines: #404041;

When building components, we could use these tokens to define colors for the component elements. In the case of a checkbox, the chosen names work very well.

Diagram showing how $dt-disabled is assigned as color to a disabled checkbox color

For a component of similar semantic meaning, we can use the same names. You can see below how the design tokens translate to the scope of a radio button:

Diagram showing how $dt-disabled is assigned as color to a disabled radio color

However, when it comes to a different component, the same names may not reflect the purpose of the color even though the color palette stays the same. Let's have a closer look at a hypothetical progress bar where we re-use the colors we had available:

Diagram showing how $dt-disabled is assigned as color to the empty space of a progress bar because for aesthetic purposes

The "primary" color works well for the progress content color. But the label text uses "lines" and the bar’s background is “disabled." These names are confusing and make the component styles fragile. We’d be in trouble if, say in a later iteration, the color of the lines is changed to a very light gray, making the progress bar label hard to read as an unintended side-effect.

It would make sense to keep color-like names for the tokens, because then they easily translate to any component. But how semantic are such names?

Meet me at the middle layer

With design tokens we are trying to create a global context for the whole interface we are building, including future not-yet-known components. No matter how thoroughly we try to anticipate all the different twists and turns app development might take, we will never cover all future bases. It may seem that we nailed it on the first iteration, but sooner or later new components will come and shatter our pristine global token names – that is unless we introduce a middle-layer between the universal design token and the variables used within the context of a component.

Middle layers are common practice in software engineering when we have to match two abstractions. In this case, we introduce local design tokens that make sense at the component context level but which receive their data from universal design tokens.

Let's transform our previous example into a middle layer approach. Doing so requires us to think differently about the semantics we provide to our tokens. In this iteration, let's define our global design tokens as follows:

$dt-color__neutral--900: #000000;
$dt-color__neutral--700: #404041;
$ds-color__neutral--400: #767677;
$dt-color__neutral--100: #FFFFFF;
$dt-color__neutral: $dt-color__neutral--900;

$dt-color__primary: #f74d7b;

$ds-color__transparent: transparent;

Notice that now instead of being prescriptive about the purpose of each color, the semantics of the variable names lean on describing the color. For instance, instead of the name $dt-disabled we use $dt-color__neutral--700.

Within a component we will not use the global design tokens directly as we did before. Rather, we'll define new component-scoped variables to serve as a middle layer. It's only at this point that we associate functionality to our design tokens:

$dt-form__border: $dt-color__neutral--700;
$dt-form__border--disabled: $ds-color__neutral--400;

$dt-form__content--active: $dt-color__neutral--100;

$dt-form__background: $dt-color__transparent;
$dt-form__background--disabled: $ds-color__neutral--400;

Using the middle layer approach we employ these variables in the checkbox component instead of referencing the universal design tokens directly:

Diagram showing how $dt-form__background--disabled is assigned as color to a disabled radio color

And because radio buttons and check buttons share the same semantics for colors, we can reuse those names for both components:

Diagram showing how $dt-form__background--disabled is assigned as color to a disabled radio color

Best of all, we don't have the problem of conflicting names for the progress bar. Instead, we can define a local design tokens that make sense for the different elements of the component while still using the overall colors:

$dt-progressbar__background: $ds-color__neutral--400;
$dt-progressbar__content: $dt-color__primary;
$dt-progressbar__text: $ds-color__neutral—700;

Diagram showing how $dt-progressbar__background is assigned as color to the empty space of a progress bar

A naming old-friend: BEM

Does the naming scheme for the variables remind you of something? It should: it's Block-Element-Modifier syntax. BEM was created to help developers write consistent CSS before contextual styles were available. Technically, in these times of CSS-in-JS solutions, we have a scope per every component, so there is no need to simulate "unique" CSS classes (which was BEM's original intent). However, given its popularity, BEM can help developers create consistent variable names for design tokens without introducing new mental models.

Following BEM in choosing the names for our variables, we can define the main entities of our system (aka "blocks"): sometimes they are the components and other times semantical groups of components (like "form"). The "elements" are the parts that comprise the components, and "modifiers" are optional variants for components.

By defining the variables this way we can set up the inner structure of the component(s) we are developing, listing all their elements and modifications. Everything becomes visible, leading to what I call “conscious development.”

The same naming scheme can work for both universal design tokens and per-component variables. As we observed above, we can describe universal design tokens with BEM's conventions around semantics:

$dt-color__neutral--900: #000000;
$dt-color__neutral--700: #40404A;
$ds-color__neutral--400: #D2D2D6;
$dt-color__neutral--100: #FFFFFF;
$dt-color__neutral: $dt-color__neutral--900;

$dt-color__primary: #006AED;

$ds-color__transparent: transparent;

In this case color is the block we're describing, neutral an element of that block, and 900, 700 are variations.

In the component variables example, we describe all the inner component elements and their states. With the form case the full list will be more extensive than before:

$dt-form__border: $dt-color__neutral--700;
$dt-from__border--active: $dt-color__primary;
$dt-from__border--disabled: $ds-color__neutral--400;

$dt-form__content: $dt-color__transparent;
$dt-form__content--active: $dt-color__neutral--100;
$dt-form__content--disbaled: $ds-color__neutral--400;

$dt-form__background: $dt-color__transparent;
$dt-form__background--active: $dt-color__primary;
$dt-form__background--disabled: $ds-color__neutral--400;

Note that the same colors translate to more than one variable. Even if two styles are visually similar, sometimes they have different meanings. Having a fine-grained semantic description for the design tokens of our components helps us decouple styles from tokens, which will enable us to execute future changes more easily when the time comes for a rebranding or a new dark theme.

Middle layers work everywhere

As you may notice, the examples in this article are given in SCSS. However, it's worth to highlight that the idea of a middle layer connecting global design tokens with the scope of each independent component is conceptually technology agnostic.

For example, the same approach will work with styled-components. We'd define our universal style tokens as follows:

export const colors = {
'neutral--900': '#000000',
'neutral--700': '#40404A',
'neutral--400': '#D2D2D6',
'neutral--100': '#FFFFFF',
'neutral': '#000000',

'primary': '#006AED',

'transparent': 'transparent',
}

And implement the component-specific tokens like this:

import { colors } from './tokens.js';

const form = {
  'border': colors['neutral--700'],
  'border--active': colors['primary'],
  'border--disabled': colors['neutral--400'],

  'content': colors['transparent'],
  'content--active': colors['neutral--100'],
  'content--disbaled': colors['neutral--400'],

  'background': colors['transparent'],
  'background--active': colors['primary'],
  'background--disabled': colors['neutral--400'],
};

const StyledCheckbox = styled.div`
  // ...
  background: ${props => props.disabled ? form['background--disabled'] : form['background']};
  // …

Because we naturally have scopes with styled-components (and JavaScript in general), there is no need to follow BEM for naming any more. However, I like to still have the modifiers for the described entities named this way because it helps keep the structure flat.

Conclusion

Re-assigning design tokens to per-component variables makes it possible to define the design tokens as global entities but develop the components operating within their own semantic context. The middle layer approach pushes designers to explicitly describe the component visual architecture and that translates to its better understanding. Such a “conscious development” approach is especially helpful when the work is shared by the team.

Unleashing the Genie: Helping both designers and developers with Storybook

Continue reading →

Interactors: the design systems testing ally

Components from a design system make building UIs easy. It should be just as easy to test them. Interactors enable more robust tests for component-based UIs, and improve component libraries' maintainability.

Continue reading →

Renaming `main` with purpose

Jorge Lainfiesta — Fri, 11 Dec 2020 05:00:00 GMT

At Frontside main (or any similar name) has never seemed like a great branch name. Like many developers, we navigate through different kinds of repositories all the time, from user-facing apps to Open Source libraries. It can be difficult to keep track of how code flows from development to its final purpose when merging our work into a codebase, making naming branches not just a matter of efficiency but affecting the clarity of the code as well.

Every project has its own goal and process, which means that merging code into 'main' has different meanings. For instance, it's not the same to merge a PR into a Jamstack blog that will immediately publish the new content as it is to merge a PR into a finance app that has to go through manual QA before getting published. The same applies to library maintenance: submit a PR to a private library with few users is quite different than to submit it to a widely adopted Open Source library.

That's why we propose naming the default branch of each repository such that it reflects its purpose. For instance, in effection—a library we maintain—the default branch is v0 because the code merged there is part of version zero, and we cut releases from it. Another example is our website: the default branch is called production because everything we merge into it is automatically published to production.

Screenshot of PR getting merged to 'production'

The author and reviewers of a PR that will be merged into production are now conscious of the aims and purpose in the code in question.

In this article, we present Frontside’s naming strategy for semver libraries and the guiding principle we use for apps repositories. We also share our ideas about changing the name of the default branch of a repository in the form of a helpful FAQ.

Semver libraries

We follow the popular Semantic Versioning (Semver) format to issue releases. Semver distinguishes three types of releases: major, minor, and patch. Although it provides a semantic framework for release names, Semver makes no comment on how each team goes about developing their libraries – there's still plenty of room to decide how you cut your releases. At Frontside, instead of curating each release manually, we tie major versions to branches so we can cut automated minor and patch releases, which ultimately leads to setting our default branch name to a major version branch instead of simply main.

Releases are often manually put together by the maintainers, who also judge the type of release they're issuing. That means that branches and releases are not coupled: their relationship is curated by the library maintainers on a case-by-case basis. This approach has proven itself effective for many projects, but is an added burden for the maintainers. It becomes especially problematic with monorepos, where several packages with codependencies have to be updated and released at once.

At Frontside, we instead tie branches to major releases: everything that is merged into a v1 branch must be releasable in a v1.x.x version. That enables us to use changesets to cut minor and patch releases with minimal effort – even within monorepos.

With our approach having a main branch is just not helpful. And it's usually confusing in other approaches too: what can you expect to find in main? the latest released version? a future version? the most popular version? not a release candidate at all? It's impossible to tell by simply looking at the name, and a lot of (unnecessary) work to dive into the code to find out.

Instead of using main as a default branch, we set our default branch to be the latest major version branch that is currently under development. This means that the default branch will change over time, and that's a good thing, because the evolving name gives contributors clear directions when navigating the codebase.

Apps

Unlike libraries, there's no widely popular way of structuring releases for apps. Apps are wildly different across teams and industries not only in how they look, but also in the way they're developed and deployed. Our advice is to think of the branch as a pneumatic tube that takes the code somewhere and to ask what is the name of that place? If putting code in that tube means that it will be shuttled to production, then naming the default branch "production" is a sensible idea because (like with our suggestions about libraries) the name offers a clear indication of the code’s purpose instead of requiring you to fish around to find out what the codebase is trying to accomplish.

Repository Default Branches: a FAQ

Why should I move away from `master`?

You'll soon become obsolete if you keep master as your default branch. The industry standard for default branches is main, so we advise you to at least move to that name if you haven't. But we think you’d be wise to shift away from generic names altogether and consider moving to purposeful branch naming.

Won’t it be annoying to have different default branch names for my projects?

Perhaps at the beginning. But it's not common for developers to push to main: in fact it's generally discouraged. Interactions with the default branch are therefore limited: typically to just when cloning a repo, creating a PR, or syncing up with the latest developments.

What’s the upside of using specifically named default branches?

Everybody working with your codebase will have a concrete understanding of the goal of the repository. From the very beginning when they clone your repo until when they create a PR, everyone clearly understands the purpose and goal of their work.

How do I change my default branch name?

Github is making it easier for you to change the default branch of your repository. Check out their recommendations for renaming your default branch.

Conclusion

Naming is hard. It's been a challenge in computer science for decades. But what we call things conditions the way we think about them. At Frontside we think it’s time to move away from the “tried and true” generic names – especially when these naming conventions are actually causing inefficiencies and unnecessary work.

MirageJS: Choosing the right Serializer

Min Kim — Mon, 31 Aug 2020 05:00:00 GMT

MirageJS is one of the pioneers in enabling frontend developers to use mocks systematically for their tests and development. It was originally built for Ember.js in 2015, but due to its popularity the maintainers created a standalone version compatible with any javascript framework.

Mirage introduces itself as a mocking library, but at Frontside we don't think that's a good enough term to refer to its extensive power. If your application uses a RESTful API (an adapter for GraphQL is on the way), Mirage is a great choice to simulate your backend with high-fidelity outputs. We've used Mirage in several projects to enable frontend developers to keep advancing the UI while the backend is not yet ready. Its flexibility also makes it a perfect test companion compatible with many tools out there, including Cypress.

A key element of Mirage that enables it to generate realistic responses lies on its serializer layer. The serializer takes the data stored in Mirage's seeded database and outputs it in the format that you configure for it. For an effective simulation strategy, you want that output format to match exactly your backend API specifications.

In this article, we'll explore the four serializers available on Mirage by default. Choosing which one is the right one for you depends on your API formats. There might be a serializer that works with your backend right out of the box or you may have to customize it from a base. After outlining how the serializer layer works in Mirage, we'll dive into the specifics of each one.

How Mirage Serializers Work

Understanding how Mirage’s various serializers work begins with reviewing how data flows thought Mirage. At the most basic level, Mirage allows you to create mocks in a traditional way, returning hand-crafted objects:

this.get("/blog-posts", () => {
  return [
    { id: "1", title: "Beyond mocking" },
    { id: "2", title: "A lesson about Interactors" },
    { id: "3", title: "Github Actions: all about pull_request" },
  ];
})

However, hand-crafted anything is not a scalable solution, and these sorts of objects become difficult to maintain when your data relies on relationships.

Instead, we recommend taking advantage of Mirage modeling and ORM capabilities so you can shape your backend in Mirage and seed it with generated data. As the snipped code below illustrates, you can use the seeded models of the database directly from your route handle:

this.get("/users", (schema, request) => {
  return schema.users.all()
})

In this case you don't have to manually deal with IDs, formats, or the concrete user records. Instead, you use a schema query that will bring records (based on models and factories) from Mirage's database and output the format that your backend uses.

In order to do that, you have to set up models, factories, and serializers in Mirage. Models deal with the relationship between records, factories generate sample data for the records' attributes, and serializers format how the records will be mapped in a JSON response. The following diagram illustrates how these pieces come together to enable you to query records in your Mirage route handles:

Diagram: Set up relationships → configure data generation → specify JSON response format through serializers → Query schema on route handle

The role of the serializer is to map the data to a JSON structure that matches your backend needs. The structure of the JSON refers to how the object keys are named and what's nested inside which keys.

Given that the serializer's main purpose is to make it easy for you to have Mirage outputting realistic data for your frontend, we recommend discussing with your backend team how they plan to format the responses from their endpoints. Then you can compare those responses to what the different Mirage serializers offer and choose the closest one, customizing it if necessary.

Model and Factories Setup

The job of a serializer is to provide a format that represents the attributes and relationships of the records stored in Mirage’s database. To illustrate how the different serialiser options work, in this section we’ll describe a hypothetical set up. It’s worth noting at the outset that in Mirage models only describe relationships, not attributes; that means attributes can be defined arbitrarily when the model is instantiated in a factory.

In the code below we’ll use three models to describe the relationships among our data:

models: {
  author: Model.extend({
	blogPosts: hasMany(),
  }),

  blogPost: Model.extend({ 
	author: belongsTo(),
	post_blogComments: hasMany()
  }),

  post_blogComment: Model.extend({
	blogPost: belongsTo()
  })
}

Notice the inconsistencies in case and separators in the names for the model’s fields. We've used arbitrary casing so the effect of the serializers described below is more apparent.

Now for the factories, we’ll set them as described below. As we mentioned earlier, it’s in this part where the attributes for our records are declared:

factories: {
  author: Factory.extend({
    author_emailAddress: () => faker.internet.email()
  }),
  blogPost: Factory.extend({
    blogPost_title: () => faker.random.words(3)
  })
 post_blogComment: Factory.extend({
    comment_paragraph: () => faker.random.words(6)
  }),
}}

Again, to highlight the effects of serializers on the attribute names, notice that in the code above the casing for each attribute is mixed. You can see we're using a library called faker to generate random data to populate the models attributes.

To make the outputs comparable across serializers, the example responses that we provide are the result of fetching a request for all blogPosts:

schema.blogPosts.all( )

Data attributes on all serializers

Before getting into the specific serializers, let’s address first a peculiarity about attributes that all Mirage serializers share. As it turns out, Mirage doesn’t serialize all the attributes of a response evenly.

When we fetch all the blog posts with schema.blogPosts.all( ) we'll get all the records for blogs posts. If we configure the serializer to include relationships, we can expect the authors and comments from such blog posts too. The following diagram illustrates the relationships between blog posts, authors, and comments.

Diagram: Blog Post has a belongsTo relationship with Author and a hasMany relationship with comments

Because we’re using schema.blogPosts.all( ) to get these blog posts, we’ll say that blog posts are the queried records. As mentioned, included in that response we’d have the author of the blog post (through belongsTo). Additionally, we’ll get the blog post comments (through hasMany).

Serialization on queried records: attribute names are transformed

Serializers will only transform the attribute names of the queried records. The transformations applied vary from serializer to serializer; you’ll find the specifics in the sections below.

Records from belongsTo: attribute names are not transformed

Counterintuitively, the serializers in Mirage will not apply any transform to the attribute names of the records coming from a belongsTo relationship. This means that the inconsistent attribute name we included in our authors factory will remain as it was declared, author_emailAddress no matter which serializer you use.

Records from hasMany: attributes are not included

Perhaps to avoid loading large amounts of data, Mirage does not include the attributes of any record that has been loaded due to a hasMany relationship. This means that although we’ll get a list of the comments associated to the blog post we queried, we will not be able to get any of their attributes.

JSON API Serializer

This serializer follows the JSON API specification. A fetch for blogPosts with this serializer will return the following response format:

{
  "data": [
    {
      "type": "blog-posts",
      "id": "1",
      "attributes": {
        "blog-post-title": "A blog post"
      }
      "relationships": {...}
    },
    ...
  ],
  "included": [...]
}

Field names format: dashes

In compliance with JSON API conventions, all field names are changed to use dashes. As you can see in the example, the names that were inconsistently cased are all dasherized now; for instance, blogPost_title becomes blog-post-title.

Response structure: fixed top level elements

There are two top level keys for the JSON API response: data and included.

The main part of the response is data, an array of entries that answer to the query issued. In this case, it's providing all the blog posts available.

The other key, included will have the record information from the references related to the entries in data. By default, included is empty on Mirage.

Relationships: reference objects

Each record on data can have a relationships key. In our example, the blog post belongsTo an author and hasMany blog post comments. By default, the relationships are not included in Mirage by the JSON API serializer.

To include the relationships, we can configure the include option in the JSON API serializer. In that case, we'll get a relationships key inside each record from data with references to what those authors or comments are.

Below you can see how a single record looks like when the relationship references are included:

{
  "type": "blog-posts",
  "id": "1",
  "attributes": {
    "blog-post-title": "A blog post"
  },
  "relationships": {
    "author": {
      "type": "authors",
      "id": "5"
    },
    "post-blog-comments": {
      "type": "post-blog-comments",
      "id": "5"
    }
  }
}

Notice that in JSON API Serializer, types are all dasherized and pluralized, and that they are used in every record. For instance, a single author has a type authors.

Additionally, all the records referenced in all the relationships key will be loaded in the included key of the response.

Alternatively, we can set the alwaysIncludeLinkageData option to true on the JSON API serializer, with which we'd get the relationship information on the data records, but those related records will not be included in the include array.

More details

Active Model Serializer

Active Model is the standard format for Rails APIs responses. Chances are if you don't already know whether or not you need to use this serializer, you most likely don't need this one. But let's still have a look at how this one handles its response. We'll be evoking the same fetch as the one we used for the JSON API Serializer. This is how the response with the Active Model Serializer would look:

{
  "blog_posts": [
    {
      "id": "0",
      "blog_post_title": "A blog post",
      "author_id": "...",
      "post_blog_comments_ids": [...]
    }
  ]
}

Field names format: under_scored

The Active Model Serializer changes the caps to be under_scored no matter how the original model name was configured. As you can see in the example, the attribute names that were inconsistently cased are all under_scored now; for instance, blogPost_title becomes blog_post_title.

Response structure: top level types

The Active Model Serializer presents the entity types on the top level arrays, where the type name is pluralized. Unlike the JSON API Serializer the data is not nested under data or included objects. You can think of the response having lists of records grouped by type on the top level. In this example, blog_posts is the type of the queried objects and it's at the top level. Relationships: _id suffixed reference (or embedded)

By default, relationships are not resolved for this serializer. However, we can configure the include option for the Active Model Serializer and then we'll receive them as references but in a different format than JSON API.

If include is configured, the records of the related fields in a relationship are also loaded at the top level of the JSON with their type names. In the target object, the reference to that related object is only an id.

The field name of the relationship will be appended _id:

{ 
  "blog_posts": [
    {
      "id": "1",
      "author_id": "5"
    }
  ],
  "authors": [
    {
      "id": "5",
      "name": "Mika"
    }
  ]
}

Additionally, you can decide to configure the embed option to nest the relationship object instead of having a reference:

{ 
  "blog_posts": [
    {
      "id": "1",
      "author": {
        "id": "5",
        "name": "Mika"
      }
    }
  ]
}

More details

Active Model Serializer Source code

Rest Serializer

The Rest Serializer is almost identical to the Active Model Serializer except that it transforms names to camelCase instead of under_score. Let's look at the example output for Rest Serializer:

{
  "blogPosts": [
    {
      "id": "0",
      "blogPostTitle": "A blog post"
      "author": "...",
      "postBlogComments": [...]
    }
  ]
}

Field names format: camelCase

The Rest Serializer changes the caps to use camelCase no matter how the original attribute name was configured. As you can see in the example, the names that were inconsistently cased are all camelCased now; for instance, blogPost_title becomes blogPostTitle.

Response structure: top level types

The Rest Serializer presents the entity types on the top level, in the same way Active Model Serializer does: the top level keys are pluralized camelCase types. Relationships: no-suffix id reference (or embedded)

By default, relationships are not resolved for this serializer. However, we can configure the include option for the serializer and then we'll receive them as references.

{ 
  "blogPosts": [
    {
      "id": "1",
      "author": "5"
    }
  ],
  "authors": [
    {
      "id": "5",
      "name": "Mika"
    }
  ]
}

Additionally, you can decide to configure the embed option to nest the relationship object instead of having a reference:

{ 
  "blogPosts": [
    {
      "id": "1",
      "author": {
        "id": "5",
        "name": "Mika"
      }
    }
  ]
}

More details

Rest Serializer Source code

Serializer

Last but not least is the barebone Serializer which MirageJS describes as basic. Its behavior is quite similar to Rest Serializer, with the only difference lying in how the relationship references are suffixed.

Let's check out the output for Serializer:

{
  "blogPosts": [
    {
      "id": "1",
      "blogPostTitle": "A blog post"
      "authorId": "...",
      "postBlogCommentsIds": [...],
    }
  ]
}

Field names format: camelCase

This serializer changes the caps to use camelCase, no matter how the original attribute name was configured. As you can see in the example, the names that were inconsistently cased are all camelCased now; for instance, blogPost_title becomes blogPostTitle.

Response structure: top level types

The Serializer presents the entity types on the top level, in the same way Active Model Serializer does: the top level keys are pluralized camelCase types. Relationships: Id suffix reference (or embedded)

By default, relationships are not resolved for this serializer. However, we can configure the include option for this serializer and then we'll receive them as references.

If include is configured, the records of the related fields in a relationship are also loaded at the top level of the JSON response with their type names. In the target object, the reference to that related object is only an id.

The field name of the relationship will be appended Id:

{ 
  "blogPosts": [
    {
      "id": "1",
      "authorId": "5"
    }
  ],
  "authors": [
    {
      "id": "5",
      "name": "Mika"
    }
  ]
}

Additionally, you can decide to configure the embed option to nest the relationship object instead of having a reference:

{ 
  "blogPosts": [
    {
      "id": "1",
      "author": {
        "id": "5",
        "name": "Mika"
      }
    }
  ]
}

More details

Serializer Source code

Cheat sheet

	JSON API Serializer	Active Model Serializer	Rest Serializer	Serializer
Response structure	Top level `data` and `included` keys. Every type-name is used in plural in each record.	Top level keys by pluralized type_name.	Top level keys by pluralized typeName	Top level keys by pluralized typeName
Casing transforms	dasherized	under_score	camelCase	camelCase
Relationships included in response?	No, unless `include` is configured. Option for `alwaysIncludeLInkageData`	No, unless `include` is configured	No, unless `include` is configured	No, unless `include` is configured
Relationship reference in record	`relationships` key with reference object. ex. `"relationships": { "author": { "type": "authors", "id": "5" } }`	Name of attribute suffixed with `_id` ex: `"author_id": "5"`	Name of attribute with no suffic ex: `"author": "5"`	Name of attribute suffixed with `Id` ex: `"authorId": "5"`

Beyond Mocking: Decoupling teams through Simulation

Charles Lowell, Jorge Lainfiesta — Wed, 29 Jul 2020 05:00:00 GMT

Working across teams is challenging. It gets especially tricky when one team depends on another to make progress on their own tasks. For instance, take an all-time classic: the front-end team needs the back-end team to provide them with APIs to advance the UI. Both parts are working towards a common goal, but their dependency may create tensions and frustration, which compromise the quality they would otherwise deliver.

In Frontside’s experience, the most effective solution to this kind of collaboration problem is to decouple the teams through a simulator. The simulator must accurately reflect what the back-end team is building, such that the front-end can rely on it without future headaches. Those developers tasked with creating the UI should be able to flawlessly switch between the simulator and the real back-end without effort.

An effective and useful simulator is not merely a collection of JSONs that reflect the structure of server responses. It also emulates errors and relationships between endpoints in order to capture the inner state of the backend service. Yet just as crucial as the simulator are the practices adopted by the team to keep it useful and reliable.

In this article, we’ll start by exploring why decoupling front-end and mobile teams from the back-end is valuable. Then we’ll jump into what a good simulator does and how it critically differs from using mocks and stubs. Afterward, we’ll outline the key steps you need to take if you adopt a simulation strategy.

Why you should consider decoupling

Implementing decoupled modules when developing software creates a more robust and resilient system. Within a decoupled architecture, specialized layers in the system are optimized, modules can swiftly be changed for newer ones, and one-point failures are avoided altogether.

In the same fashion, decoupled teams create a flexible and robust organization. More often than not tightly coupled teams lead to low productivity and high frustration. When you force two (or more) teams to be dependent on each other, you are potentially wasting the energy of both and ignoring their unique challenges. Each team has its own people with their own capabilities. Successful managers recognize each team’s virtues and use them to improve the team’s performance and morale. However, leveraging the skills and motivation of each team becomes difficult when two different teams become each other’s blocker. In that situation, what could have been synergy becomes friction.

When you run tightly coupled teams, it’s not simply that problems in which each team runs into impact both teams, magnifying each delay and obstacle and impacting the organization as a whole. It’s that in not addressing the issue you essentially communicate that you don’t respect each team’s most valuable asset: time. From a project management perspective, teams reap considerable benefits from decoupled cadences when they run into bottlenecks or have a disparity in their respective priorities.

For example, let’s say new data protection practices are set to be adopted in the organization. The back-end team then has to prioritize the revision of a large set of endpoints and halt all forward progress on new features till data protection issues are resolved. The new requirements don’t show up in the UI so the front-end team doesn’t have to deliver anything. They’d like to continue working on the UI for the new features, but now are stuck because the back-end team is unavailable and thus cannot deliver the APIs that the front end team needs to tackle to complete its tasks.

In cases like these, it would be easier for both teams to run at their own speed according to their particular needs and requirements instead of strictly forcing both to adhere to an artificial cadence so that they can stay coupled. Respecting the time of both teams means the correct solution is decoupling.

What is a simulator?

No discussion of a simulator is complete without reference the film The Matrix, so let’s start there. The movie shows a world in which humans live in a computer simulation that is very different from their actual physical existence. This simulation is not merely a collection of inert pictures flashing before their eyes—an interface that you passively interact with. It is instead a fully immersive experience that you actively engage—a world so alive don’t notice you’re within it because it feels real.

Morpheus, from the movie The Matrix (1999), sitting in a couch

Morpheus, from the The Matrix (1999)

To truly be effective, the simulator software developers use has to be alive and dynamic like the Matrix. Indeed, that’s the problem with mocks and stubs—no matter how realistic they look, in the end they are like two-dimensional photographs—dead and static. Contrary to mocks and stubs that decay rapidly and become useless, a simulator can be updated and stay evergreen.

The proof is in the pudding as they say, and all you need to do is return to mocks or stubs that were written only a month prior. Being hard-coded they become obsolete fast, and after 4 weeks feel rickety and not up to the task. While they may help hold things together for a short time, so does duct tape – neither is a long-term solution. Mocks and stubs instead rapidly become a problem for the codebase.

State and relationships bring life to simulators

We have described mocks and stubs as dead vs. simulators as alive, but to what does that translate to in technical terms? The key lies in state and relationships. Mocks and stubs merely reflect the response structure of an API response, while simulators map the APIs behavior and relationships and from there outputs appropriately formatted responses.

That turns out to be a huge difference. Real APIs are not static JSON objects independent from one another like in a mock or stub. APIs interact among each other, depend on each other, and even consume each other in the back-end. Moreover, each endpoint has errors, tokens, metadata, and side effects. None of that is reflected in a static mock or stub. And after all, how could it? JSON is merely the format over which all the information from the back-end is communicated. The format is not a crucial part of an API, and therefore it is not uncommon to see it change and evolve. But that means the mocks and stubs become quickly outdated in addition to lacking the dynamic character of real APIs.

Screenshot of Network tab on Firefox's Web Inspector

Focusing on the responses' format is important, but it's helpful to be aware of what it represents in the back

A simulator is designed to accurately represent the API’s inner state, which is reflected in the interactions and relationships between endpoints. For instance, if you edit the user’s name through the UI, the endpoint that returns the user data would reflect such change, although it was introduced as an effect of calling a different endpoint.

With mocks and stubs, you can replicate the structure of a response provided by an API. It may even appear suitable for simple features. You just add a file with a JSON to your codebase and consume it as a response for your API request; no annoying extra libraries or setups. But the problem is that APIs seldom are simple. Sooner than later, you’ll be dealing with authentication tokens, form validations, file uploads, and more. API complexity scales up rather quickly in most applications. That’s when you see the real value of simulation.

Adopting a simulation strategy

Opting for a simulation strategy will directly impact your overall development process. That’s why a simulation strategy must be a team decision and effort. You have to invest in understanding your back-end, research libraries for simulation, implement the simulator, and maybe most important of all, socialize and maintain the simulator’s accuracy.

The first step is understanding how your back-end is provisioned. You might find yourself getting into a rabbit hole when exploring how different teams in your organization provide their services and how they interact (or not) with each other. Broader issues may often come up, such as the opportunity to consolidate the services architecture with better-aligned practices across teams or through a federated schema.

The next priority is researching and then choosing the most suitable simulation tool. For back-ends over HTTP, we’ve had good experiences with Mirage.js. But you might find out that it’s not sufficient to cover your use cases, so you may need to invest in extending tools or even creating your own (beware: this introduces greater risk and maintenance costs in your simulator). When a simulator doesn’t exist for what you need, you can consider cooperating with open source projects.

Once you have decided on which libraries and tools and have a sufficient understanding of your back-end, you can start implementing your simulator. We recommend approaching the task from an experience-first perspective instead aiming to achieve endpoint parity. Focus on a UI segment that you know well and simulate the endpoints that it needs. Try experimenting on how responses with errors, latency, and other parameters look like. It will most likely take a few iterations before you develop to the best patterns to follow for all the APIs you want to implement.

Finally, socializing simulation practices is essential for maintaining its reliability, accuracy and effectiveness. Developers will have to extend the APIs according to their needs. But because those needs may be redundant (or in conflict) with other data requirements, an effective process for extending the data schema is needed. That’s why the simulation adoption must include a comprehensive plan for discussing and curating the evolution of APIs to keep the simulator evergreen. This benefits not only the simulator but the sustainable growth of the organization’s data management. This process is typically overseen by one or more developers who takes on the role of data field owner.

Conclusion

Implementing a simulation strategy takes resources: very tangible time and money. It requires developers to research, experiment, and adopt new practices. It’s not easy to come to your boss and say, “Hey, I’ll be working on a simulator, so don’t count on me. See you in a month or two!” Because simulation has concrete costs and requires specific resources, it is easy for decision-makers to take that number, compare it with other expenses, and even reject it as an option because it is a “non-feature.”

But what is often overlooked in that decision is the cost of the solution used instead of a simulator. While not nearly as quantifiable as building a simulator, the cost is real, and quite arguably much more than what it would take to adopt the simulator solution. To be sure there’s the technical debt costs of the multiple mocks and stubs accumulating untamed in the codebase, but both literally and rhetorically you can’t put a price tag on the lost time and productivity due to blocked teams. And as the project scales up, there’s the problem of the need for automated tests and slow or even flaky results if run against remote back-ends. Creating ad-hoc mocks and stubs to “get through” the features only results in long-term liabilities in your codebase.

Or you could address all these issues in one fell swoop by using a simulator. To be sure, adopting a simulation strategy requires thoughtful considerations and efforts from the team to adopt new practices. But the benefits that flow from decoupling teams through simulation – more effective development cycles and a better directed collaboration – by far outweigh the problems caused by mocks and stubs.

In The Matrix the main character Neo initially doesn’t trust the simulation but eventually comes to recognize the power inherent in using it to achieve his ends. Maybe it’s time for you to “take the red pill” like Neo did and convince your company to do the same. Feel free to reach out to us if you want help or ideas :)

The Lesson of BigTest Interactors: never write a flaky test again!

Jonas Niklas — Thu, 16 Jul 2020 05:00:00 GMT

Testing the frontend of complex applications is often associated with nightmarishly inconsistent test suites, where seemingly random failures cast doubt on the value of the tests themselves and frustrate developers to no end. Our mission with BigTest has been to eliminate flakiness in tests altogether and provide the most stable, easy to use, and best performing test framework out there.

As the original author of Capybara, an acceptance test framework written in Ruby which has a reputation of being incredibly solid and resistant to flakiness, I know all about how to avoid random test failures. We have taken all of the lessons learned from tweaking Capybara over years and applied them to build BigTest’s Interactor API—a powerful way of defining interactions with your application which allows you to write tests which are even more solid than what Capybara can provide.

We'll be ready to share more about BigTest soon, but for the moment I want to talk about the overarching approach Frontside takes to eliminate flakiness, and what you need to know to write tests which are rock solid and never fail randomly. While there will be significant gains to be had in using Frontside's BigTest, the principles guiding its development are what I want to talk about here: eliminating ambiguities when writing tests in order to ensure stability. Let's see how BigTest is designed to do just that and then apply those insights to test writing in general.

The Convergence Strategy

The approach BigTest uses is sophisticated in its execution but fundamentally simple in its design: for any interaction we want to perform, or any assertion we want to check, we try it and see if it succeeds; if not we wait and try again. We keep doing this until we either succeed, or some set amount of time has expired and declare the test a failure.

We call this strategy "convergence" and functions which exhibit this waiting behavior “convergent." In order to create convergences in a web UI, we must be able to keep reliable references to the elements we want to act and assert on, which is possible in BigTest through having control over the event loop of the browser.

Retaining references

The most important detail is that an interaction or assertion must fully capture all of the work that it needs to perform and perform all the same work again and again in the retry loop. The test in other words must run on the same content each and every time. To ensure this, BigTest never uses any outdated references to elements, and everything is always run against the actual DOM.

For example, let's imagine we have a field for a name and this field is within a form. With BigTest interactors we could set the value of this field to look like this:

await Form.byId('login').find(TextField.byId('name')).fillIn('Jonas');

We can think of this as roughly performing three steps under the hood:

let form = document.querySelector('#login'); // step 1
let textField = form.querySelector('#name'); // step 2
textField.value = 'Jonas' // step 3

Now just re-running step 3 over again and again is not enough. The form element or text field could change, and either of those changes could cause us to retry something which would never work. In order to make this as stable as possible we need to re-run all of the steps every time.

A very simplified algorithm for how this would work looks roughly like this:

while(true) {
  let startTime = new Date();
  try {
    // Run our actual interaction code
    let form = document.querySelector('#login'); // step 1
    let textField = form.querySelector('#name'); // step 2
    textField.value = 'Jonas' // step 3
    // if it fully succeeds, break out of the loop
    break;
  } catch(e) {
    // if there was an error, check if we've already been trying for too long
    if((new Date() - startTime) > MAX_WAIT_TIME) {
      throw e;
    } else {
      // wait a little while
      await sleep(1);
    }
  }
}

Now what BigTest's API interactors do is automatically transform the following much simpler instructions into the above algorithm for you:

await Form.byId('login').find(TextField.byId('name')).fillIn('Jonas');

Of course there are a number of details that are omitted in the simplified code above. For example, each interactor must be specific enough to match a single element on the page. If there were multiple fields with the id name then the interaction above would fail. Designing BigTest this way forces the writer to be specific so that they don't end up interacting with a random element on the page. Otherwise layout changes where the order of elements is altered might lead to broken tests.

Taming the event loop

One huge advantage that BigTest has over Capybara is that since we are running tests directly in the browser, we can leverage the event loop and ensure that our interactions run fully within the same tick of the loop. This eliminates quite a number of problems that plague Capybara and similar Selenium-based frameworks and tools. By running within the event loop and ensuring that we never release it, we can guarantee that our elements are never stale.

Frameworks that do not have direct access to the event loop often get errors because the elements they were looking for stop “existing” for one reason or another. This loss of reference could happen because the user, the browser, or an external script triggered a change that made the reference disappear. Capybara goes to great lengths to avoid the dreaded StaleReferenceError, but BigTest simply avoids the issue altogether. That is because getting a hold of the event loop guarantees that we can run assertions when elements in the page are ready, and before they are changed by other elements.

The Interactors API reflect their event-loop synced nature. Consider, for example, a TextField interactor:

const TextField = createInteractor<HTMLInputElement>('text field')({
  selector: 'input',
  locators: {
    byId: (element) => element.id,
  },
  actions: {
    fillIn: (element, value: string) => { element.value = value }
  }
});

In most test frameworks, actions such as fillIn, would have been asynchronous. But in BigTest both the locator and action functions are synchronous, which means that they do not release the event loop. This allows us to transform them into convergent functions, which can find the element and perform the action using the same reliable reference.

BigTest will also ship with a rich library of interactors already built in, for all of the most common interface elements in web applications, so most users never have to define their own interactors if they don't want to. And yes, all our interactors are fully typesafe because BigTest is written in TypeScript.

Waiting is not enough: flakiness through ambiguities

We have claimed that our convergence strategy is enough to eliminate flakiness, but as we mentioned as well, having a solid strategy is only part of the solution, the other half of the solution is writing the tests in such a way that they make maximal usage of this solution and avoid inherent ambiguities that we cannot otherwise resolve.

Let's illustrate what we mean by this with an example: Imagine we have a paginated list with a 'Next' link at the bottom. Whenever we press the next link we will load a new set of rows, and we also update the href attribute of the next link to point to the next page.

Image of a page showing a list and "Previous" and "Next" buttons

Now imagine a written a test written like this:

await Link('Next').click();
await Link('Next').click();

Which page will we end up on? Intuitively we would say that we should end up on the third page, but unfortunately the actual answer is "it depends". The first click on Next will start a request to the server which will complete after some time, and this will change the href of the link. But if our second click happens before the request completes, we will navigate to the second page twice, and we actually end up on the second page.

A naïve solution: Guessing

A common way to work around this issue is to try and guess when an interaction is "done" – for example, by checking if there are any active requests to the server and waiting for those to complete. This was the original strategy used in Capybara, but we realized that this strategy has a fatal flaw and moved away from it. Here’s the problem:

Let's imagine that the event loop is released between the request completing and the DOM being updated. Our second click could now occur in this small window of time. What we have done is made the window of time for flakiness to occur smaller, but we have not eliminated it. This still might seem like a win on the surface, but in fact making the window smaller is actually worse, because it means that this issue will occur more rarely, and therefore in actuality will be harder to track down and fix.

A better solution: Anchoring

A better solution would be to "anchor" the second interaction by adding an assertion which ensures we have transitioned to the second page beforehand:

await Link('Next').click();
await ListItem('This item is on the second page').exists();
await Link('Next').click();

We can now be sure that we have transitioned to the second page. It is unlikely that the modifications to the DOM of rendering the second page and updating the link will occur in separate ticks of the event loop, so most likely this will be enough to eliminate the problem.

The best solution: Fixing the UI

But there's an even better solution. The problem stems from the fact that the UI itself is ambiguous. If the user of the application did hit the 'Next' link twice in rapid succession, what would happen? The result is as equally undefined as the result of the test is.

If we fix the UI to disable the 'Next' link while the request for the second page is loading, we don't need to change our test at all. When the request completes and updates the href, it will also enable the link again, and since we will keep retrying to find and click the link, we will then see the newly enabled link and click it.

By making our UI less ambiguous we have also made our tests more stable. We have effectively discovered a flaw in our UI and fixed it, made our application better, and fixed our test in the process.

Of course we don't always have control over our application or the ability to make this type of tweak. In those cases "anchoring" is a good alternative.

Conclusion

Strategies like these are why BigTest is going to be such a significant step forward in test design. BigTest’s Interactor API will make it as easy as possible to write stable tests, but in the meantime we are committed to providing explanations like these to help you understand how to achieve similar results.

Jonas Nicklas is the original author of Capybara, Ruby’s most popular framework for testing web applications. He is based in Sweden and has been working on BigTest since 2019

Github Actions: a deep dive into pull_request

Min Kim — Tue, 26 May 2020 05:00:00 GMT

Github Actions is an exciting feature that enables teams to construct workflows based on webhook events. It unlocks new possibilities for teams that neatly integrate it into their development flow.

For example, you can use Github Actions to automatically run tests, generate a preview app, and send a follow-up notification every time somebody opens a pull request. Presuming you're familiar with writing Github Actions workflows, let’s take a closer look at some aspects of that process.

At Frontside we use Github Actions extensively to make the development process of our clients both simpler and more inclusive for non-technical stakeholders. This is possible because Github Actions offers a rich API that allows us to build flexible and highly-specific workflows.

In this article, we have put together specific behaviors and information that you’ll need to use pull_request as a trigger for your Github Actions workflow. First we’ll take a look at how the workflow is actually triggered on: pull_request; then we’ll examine the information provided to the workflow when triggered by a pull_request; and finally we’ll look at how @actions/checkout reacts to pull_request.

How do workflows trigger on `pull_request`?

When a Github Actions workflow is configured to run on pull requests, by default it will be triggered on three different types of activities: opened, synchronize, and reopened.

name: Pull Request Workflow
on: pull_request

Basic pull_request usage.

If you need your workflow to run on any of the other types you will need to specify them. And if you wish to run those types in addition to the default types, you will need to manually add those too because once you configure types the default values are no longer provided:

name: Pull Request Workflow
on:
  pull_request:
    types: [ assigned, opened, synchronize, reopened ]

We have to explicitely set every action type if we customize them.
The defaults will not be preserved automatically.

Although the names of these types are pretty self-explanatory, we have gone ahead and double-checked each one of them to confirm our assumptions:

Type	Description
`assigned`	a user is assigned to the pull request
`unassigned`	a user is unassigned from the pull request
`labeled`	a label is applied to the pull request
`unlabeled`	a label is removed from the pull request
`opened`	pull request is created
`edited`	title, body, or the base branch of the PR is modified
`closed`	pull request is closed (as opposed to merged)
`reopened`	closed pull request is reopened
`synchronize`	commit(s) pushed to the pull request
`ready_for_review`	pull request is taken out from draft mode
`locked`	pull request is locked
`unlocked`	pull request is unlocked
`review_requested`	request a user for review
`review_request_removed`	remove request from user for review

For a complete list of activity types, checkout Github's Events-that-trigger-workflows page.

One thing to note is that when you configure your pull_request workflow to be triggered on labeled or unlabeled and you add/remove multiple labels at the same time, the event of each label will trigger its own workflow run. It also works the same way for assigned/unassigned and review_requested/review_request_removed.

What information is available from a workflow?

You can find the payload of each run of a workflow inside workflow/event.json which is accessible with the github_event_path environment variable. The payload contains the POST data of the webhook event that triggered the workflow, such as the action type, the commit before and after the action was triggered, the sender, etc.

Here is an example of a payload from a workflow triggered by pull_request on synchronize:

{
  "action": "synchronize",
  "after": "abcdefg", // most recent commit of this push
  "before": "1234567", // latest commit prior to this push
  "number": 12, // pull request number
  "pull_request": {
	// pull request metadata
  },
  "repository": {
	// owner, description, git_url ...
  },
  "sender": {
	// information of user who triggered this event
  }
}

You can do all sorts of useful things with the data provided from this payload. For example, you could compare the repo.url property of pull_request.head and pull_request.base to determine if the pull request was created from a forked repository against the original, and if so you could console.log("Thank you for your interest in our open source project!").

The information in the payload will vary depending on the webhook event and the activity type which caused the trigger. By way of comparison, here’s the payload from a push webhook workflow triggered by the same commit as the example above (note that the actual payload will have more information):

{
  "after": "abcdefg",
  "before": "1234567",
  "commits": [...],
  "forced": false,
  "repository": {...},
  "sender": {...}
}

As you can see the payload is different from that of a pull request. On push we have less information, namely we lack activity, number, and pull_request from the payload. Note that even if you are pushing this particular commit to a pull request, if the workflow was triggered on push, it will not have any of the pull_request properties in the payload.

Now that we have compared how a push payload differs from a pull_request payload, let's take a closer look at how the payload varies on different activity types of pull_request:

opened: Here is an example of a payload that does not have anything specific to its activity type, so the properties you see will be available on all pull request payloads:

  {
	"action": "opened",
	"number": 10,
	"pull_request": {...},
	"repository": {...},
	"sender": {...}
  }

edited: This activity type offers information on the property of the pull request the user has edited.
synchronize: As seen above, this activity includes before and after commit SHAs. The synchronize activity type is triggered per-push, so if a single push has multiple commits, the workflow will only run once. The before pinpoints the most recent commit prior to the push, such that if you have commits A-B-C and you push D-E-F, the workflow will be comparing commit F to C.
labeled and unlabeled: These include the metadata of the label that triggered the workflow:

  {
	"action": "unlabeled",
	"label": {
  	"color": "red",
  	"default": true,
  	"description": "Something isn't working",
  	"name": "bug",
	}
  }

assigned and unassigned: These are similar to labeled/unlabeled and the workflow will include the payload data of the added or removed assignee.
review_requested and review_requested_removed: as odd as it seems, neither of these provides the data of which reviewer changed, and so the payload structure for these actions is identical to opened. You are able to find the list of reviewers from within the pull_request portion of the payload, but it does not explicitly tell you which addition or removal of a reviewer triggered the workflow.

How does `pull_request` affect `@actions/checkout`?

When you use pull_request, @actions/checkout will perform a git checkout to the github.ref environment variable. Note that git checkout is not applied to the commit, as it would have been the case when using push.

This difference means that a pull_request workflow ref would look like refs/remotes/pull/##/merge whereas a push workflow would be refs/heads/branch_name. This explains why the SHA of a push workflow matches the commit that triggered the workflow, whereas the SHA of a pull_request workflow does not; instead the SHA of the pull_request is the resulting commit that was created from merging the base to the head.

You could configure the checkout action with the ref argument to be more explicit with what you want to check out. For instance, if you want to checkout the head commit instead of using the default merge ref, you could pass in github.event.pull_request.head.sha as the argument (or github.event.pull_request.head.ref if you are using v1):

- uses: actions/checkout@v2
  with:
    ref: ${{ github.event.pull_request.head.sha }}

Checking out head commit using checkout@v2.

Opting out from the default checkout to specify the head ref might be useful for any number of reasons. For example, if you have an action that relies on the output of git log, the merge commit would offset your results. Or merging with the base ref might introduce features that conflict with the head and possibly even go unnoticed. A third reason could be as follows: suppose your tests are comprehensive and catches those conflicts. A user might be confused as to why the tests are failing in CI but passing locally, especially if they are not familiar with how the checkout action works. Specifying the head ref allows you to have more granular control. In any case, a better understanding of how the checkout action works will allow you to troubleshoot effectively.

Did we miss anything? Reach out via Twitter and we'll gladly add it to the discussion! And let us know what cool things you're doing with Github Actions through Frontside’s Discord community (where we hang out and talk about these topics).

The Triple Threat to Testing — Part 2: Reliability

Thu, 30 Apr 2020 05:00:00 GMT

Imagine you have agreed to carpool with a friend. The first couple of weeks it’s all good: you go to their place to pick them up, the next time they pick you up. But one day you’re waiting in front of your building and it’s getting late. You decide to call your friend and hear them say: “Oh, I’ve been waiting too. I thought it was your turn.” You’re both frustrated with each other, and if this happens a few more times, you know you’ll drop carpooling entirely. But since you want to reduce your carbon footprint, you come up with solutions to try to make it work. Perhaps you set up a shared calendar, or create alarms on each other’s phones, or create a text message strategy. Even if your strategies aren’t foolproof, there are actionable steps you can take to significantly increase the reliability of your carpooling partner.

More often than not, test suites are like that unreliable friend. Almost every developer has encountered a test that passes several times and then unpredictably fails without having made any change in your codebase. It turns out test flakiness affects practically every team, even those with robust test suites. For instance, in 2016 Google reported an average of 1.5% of flaky tests on their suites despite their constant efforts to fight them. And test flakiness remains one of the top concerns of developers according to The State of Testing 2019 survey.

The level of trust that your test suite brings to you and your team is directly proportional to its value. When your test suite is reliable, you can move forward confidently after you see all the green checks in your Pull Request. There’s no need for you or your colleagues to manually check the whole application each time someone makes a change. That lets you iterate faster and focus the conversation on improving the quality of your app instead of wondering if the new code works at all.

Conversely, once flaky tests start to become normalized, it’s a short step to the health of the test suite progressively being abandoned. Developers begin to view writing tests as a burden that makes them less productive. But if test unreliability becomes normalized, teams wind up treating as unimportant the thing the suite was testing for. Developers thus risk putting less attention on the potential issues that the test suite should be discovering—issues that were important enough to check for in the first place.

The impact of flaky tests is not unlike the sensation of being stuck back in high school on an endless treadmill of ultimately meaningless tests. The problems run deeper than simply the lost time having to re-run tests after one fails to deliver an actionable outcome. It’s built into the DNA of developers to be problem-solvers, but flaky tests causes everybody to frustrated and annoyed at just another pointless exercise – and that robs your team of critical momentum and energy that could be directed at productively solving the issue.

What’s causing flaky tests?

All developers have experienced flaky tests. But understanding why tests are unreliable goes a long way to solving the issue. Flaky tests can be diagnosed, handled, and in many cases prevented. In this section, we’ll have a look at common sources of flakiness and strategies to make tests less prone to be unstable.

Complexity and test size

Web applications are part of sophisticated systems. Long gone are the times of <marquee> where the web was composed of single html files with table-based layouts. Even the now trendy simple-looking JAMStack is backed by advanced infrastructure. But with the ever-increasing complexity of codebases comes difficulty in determining precisely why a test is failing to produce a result. An end-to-end test will touch on many layers of an application, and while a positive result means that the application is working, a negative result might mean anything from a dropped internet connection to the server restarting.

The bigger the test—in the sense of how much of the application stack it touches—the more unpredictable it becomes. For example, several routes to an e-commerce application will be visited and large portions of the code exercised in executing a test that evaluates the whole shopping experience. Compared to a small test that checks if a single button works, the uncertainty introduced by a big test like for the shopping experience is orders of magnitude higher. Don’t take our word for it: Jest Listfield at Google found a strong correlation between test size and the resources required to run it with flakiness.

Right now you can do the following in order to manage non-determinism caused by complexity:

Prioritize which results matter most to your team: if you work on a frontend team, you most likely won’t get much value of tests failing over network errors or QA environments set up by other teams. Instead, you may find more valuable assessing that your work is correct, even if that implies opening the risk of a ‘false positive.’
Coordinate testing strategies with other teams: If the back-end team has a great test suite that covers their endpoints, the front-end team should focus on tests that do not touch on the real back-end. Bigger tests that involve both ends can be maintained independently, so their inherent non-determinism doesn’t interrupt the other end team.
Simulate parts of your stack: Simulation is a great way of removing complexity from your test without compromising a realistic outcome. We have had good results by investing in simulators for RESTful and GraphQL back-ends as well as Bluetooth devices.

Asynchrony in the app

Web applications put together many moving pieces: browser renderings, APIs loading, events triggering, loggers recording, and more. It’s therefore not a trivial thing to be precise about when to run an assertion for a test. Sometimes you wait too long, other times not long enough. In both cases we not only compromise the speed of our test, but we can end up having a flaky test suite.

It is at the level of browser asynchrony where testing tools often promise “flake free” experiences. Testing tools do a great job at detecting when changes are done in the DOM, but have a harder time dealing with pending promises and relevant network requests. It’s not surprising when we eventually get non-deterministic requests within the UI.

Right now you can do the following in order to reduce flakiness induced by asynchrony in the app:

Avoid fixed waiting times: if you have any wait(500ms) in your test, you’re bounding your assertions to something that may fail. Oftentimes using wait is a desperate measure that slows down test suites and leads to non-deterministic outcomes.
Investigate poll-based waiting: Consider instead waiting until a specific criterion is met like a selector appearing on the DOM. For example, in Capybara, you can find an element and provide additional information on what exact match you’re waiting for, while TestingLibrary provides mutation-based waiting through waitFor.
Analyze API request waiting options: HTTP requests are one of the most significant sources of test flakiness, but how to handle it depends on how you’re setting up the API layer in your tests. For example, if you use Cypress to simulate requests, their library provides wait(alias), which allows you to wait for an HTTP request to be ready instead of having to rely on UI effects to know if the request is done loading.
Opt for framework-native waiting: Check for smarter ways that your library offers to wait. Some tools, provide functions that reach into internal framework functionality to achieve more efficient waiting. React offers act, while Ember resolves render smartly by default with await.

Environment and Preconditions

Let’s say you’re looking to find the new hip café in a part of town you’re not familiar with. Your phone’s navigation is not behaving now: time for some old-school navigation. You ask a seemingly friendly person for directions. “Oh sure, turn left in the next block, and you’ll see it.” You easily find the café. But if you were not standing at the location where you were given the directions, “turn left in the next block” would have led you somewhere else entirely. Similarly, if a test executes actions starting from a different initial state, its results will most likely differ from what you intended. This is a common cause for flaky tests: non-deterministic initial states, often caused by state changes leaking from other tests.

Ideally each test should be self-contained, as you want to be able to get the same results from tests regardless of the order in which you run them, or whether you run all your tests or just a subset. But changes in the application state tend to slip through the thinnest cracks: databases, local storages, application data, sessions, and other forms of data persistence.

Right now you can do the following in order to reduce the chances of non-determinism caused by polluted preconditions or global variables:

Determine if you have interdependent tests: Look for tests that require being run in a particular order (deliberately or by chance) as this kind of interdependence may cause headaches sooner than you think. Reconsider if you can write your tests in a different way so each of them is self-sufficient.
Experiment with a randomized order: Some tools allow you to run your tests in a random order to test the isolation of your tests. For example, in Jest you can configure a testSequencer to run tests in a random order.
Reset your data sources: Find out what’s the most reliable (and fast) way to return your data storages to a default state after each test. For instance, Jest can monkey patch modules so that if they have shared global state they won’t leak between tests; in Ruby on Rails you can use a DatabaseCleaner to reset the state explicitly.

Unmanaged tests

Flaky tests are like invasive weeds in your garden. They will keep popping up despite your best efforts. Yet if you stop plucking them out regularly, they threaten your precious vegetables because they will consume many resources. Similarly, if you don’t manage your flaky tests effectively, they will accumulate rapidly and compromise the entire test suite’s value. For this reason, it is important for your team to have an effective process to keep tests effective and flake-free.

Of course if unreliable tests are so damaging, why haven’t more resources been poured into making them more effective? The main reason is that flakiness is a symptom of other problems in the application and test suite. Flakiness can come from application code, dependencies, environments, or tools. Figuring out what’s causing the issue can be time consuming, which it is common to “temporarily” sweep issues under the skip rug. But dead tests silently increase the test suite liability, and without a defined process on how to handle skipped tests, the opportunity to improve the app is lost.

Right now you can do the following in order to manage your test suite:

Automate the identification of flaky tests: Some companies like Google and Facebook automate detection by logging every single test run in their CI system so they can flag tests as non-deterministic. While useful, it is unlikely that this strategy alone can help during active development or when preparing a Pull Request. For this reason, Spotify introduced FlakyBot, which developers can invoke in their PR to generate a report on the reliability of the test suite.
Handle flaky tests as issues: For instance, when you detect a test is non-deterministic, you (or your bot) can create a ticket explaining what is known about that test, and flag it in your test suite as it may be necessary to skip it during the normal test runs until it is fixed. Your team should then develop a plan to deal with tests that have been reported as flaky, checking on such open tickers at least quarterly.
Learn and Prevent: As you encounter more flaky tests, your team's understanding of what causes them grows. Prevention is the distilled result of managing your tests methodologically, so document the findings that come out of dealing with flaky tests to identify the patterns used by the team in a specific area or to flag a certain technology as unreliable.

Carpooling is a great solution when it’s designed to work, and at Frontside we think that everyone should feel the same way about tests. By checking for complexity, addressing asynchronicity, investigating preconditions, and managing their tests, developers can effectively strategize to minimize the impact of flaky tests and return them to their rightful status during the development process as a useful tool rather than an obstacle to be overcome.

As The Coding Gang says, flakiness is a war you can't win, and non-determinism will appear from time to time. But we think teams can work to make these scenarios the exception rather than the rule, and learn valuable lessons when non-determinism appears to minimize the impact of flaky tests moving forward.

Up next...

Like speed, reliability is essential to a test suite, but you need to address all three legs of the testing threat in order to develop a balanced response. That means we need to talk about the final essential make-or-break factor in testing: relevance. With limited tooling to manage it, relevance relies almost entirely in human judgment, but there are steps you can take to avoid wasting resources in tests that don’t yield any value for your team.

At Frontside, we think effective testing is vital for successful software teams. We can help your team develop testing best-practices so they can deliver features confidently. Give us a shout if you’d like to work with us.

7 reasons to write good Pull Request descriptions

Taras Mankovski — Wed, 15 Apr 2020 05:00:00 GMT

In large open-source projects writing a good description in a Pull Request (PR) can be the difference between your changes being merged or ignored. These projects essentially necessitate the creation of PR descriptions, as development practices are strictly enforced by tooling and the culture of collaboration. These practices dramatically help improve the quality of the overall project.

But in closed source enterprise projects, the pressure of deadlines and insufficient or inconsistent development practices limit the attention developers can spend on crafting descriptions for PRs. It is not unusual to stumble across private repositories with numerous PRs that have no description for them whatsoever. Part of the problem is that most development environments treat Pull Requests as merge demands rather than a genuine request like they are in the Open Source community.

Yet adopting some open-source practices around PR descriptions would be a good thing even in closed environments. Without a meaningful explanation of why or how the code changes were introduced into the code base, unpleasant effects like rubber stamping, unnecessary abstractions, and knowledge silos are introduced. As such mistakes weave their way through the development process, they add up and eventually compromise the long-term success of an organization’s software.

For me, descriptions are a critical part of Pull Requests. In case you’re not fully convinced of the value of Pull Request descriptions, let me give you a fistful of reasons why you should never merge a PR without a good accompanying description.

1. They capture context

By definition, a Pull Request is a proposal to change code in a particular way. If your change is correct, then your code will likely live in the application for a long time. You will always be able to go back and look at the result of merging the pull request. But what you will not find in the source code is why the change was made. What prompted the change? Were you under pressure to fix a bug? Was this change a team decision that you implemented? All of this information is context. It provides the answer, and sometimes is the only defense against the inevitable question that gets asked: Why on earth is this code even here?

A PR with no description has no context

A Pull Request with no description lacks important context. For example, why did this PR change 53 files at once? Refactoring? New feature? Hard to tell at a glance.

2. Descriptions prevent unnecessary code

Every line of code increases your organization’s liability. Even your best code will become someone’s WTF moment. Think back to the last time that you swore at a piece of code, and moments later realize that you are the author. The best PRs remove source code. The second best are those that never get merged because they turn out to be unnecessary. But without a description such unmerged PRs are likely never to happen, resulting in code that is heavier than it needs to be.

For example, suppose you want to introduce an abstraction to the code that already exists but you were not aware of. You’d be introducing unnecessary complexity by adding a code path for something that already exists in the source code. If you don’t write a pull request description that explains what your abstraction does, you’ll miss out on the opportunity to learn that a similar abstraction already exists. Even a few un-merged pull requests per quarter can add up to a lot of code that you don’t need to worry about and maintain down the line.

3. Descriptions identify the right change increment

All programmers strive to write great code, but often this leads to making (a lot) more changes in a single Pull Request than originally intended. If you think this doesn’t apply to you, try making a bullet point list of all of the changes that you introduced in a recent Pull Request. You’ll quickly find that you made numerous changes after running across mixed whitespaces, improperly formatted blocks of code, or a missing null check that were not relevant to the main concern of your PR.

These kinds of side changes are surprisingly devious. They can slow down code reviews because your changes were unexpectedly more controversial than expected, introducing bugs that you didn’t even think of. Yet without a PR description they are fiendishly difficult to undo. By introducing the practice of describing the changes that your PR makes, you force yourself to reflect if all the changes you are making should be part of the same Pull Request. When your description has a big list of changes, it often alerts you to the fact that the PR should be split up into (several) smaller ones and merged separately.

4. Descriptions lead to better solutions

Especially on large projects it’s not possible to always come up with the best available solution on your own. In most cases there will always be someone who will know an area of source code better than you. In other cases you are that person for a particular stretch of code, yet others are making imperfect changes to it. Given the constant churn and evolution of the programming ecosystems and it’s not hard to see why it’s simply impossible to always coming up with the best possible solution when making a PR. Yet it is also in everyone’s interest that others have an opportunity to suggest alternative and possibly better solutions to what was suggested in a PR. Without a description that draws the reviewer’s attention to the approach, you’re significantly reducing the chances that a better solution will emerge through code reviews.

Recording of a long discussion on a PR

A good description can lead to great discussion on a PR. I recently experienced it when contributing to Ink.

5. Descriptions makes reviewing easier

Code reviews reduce the number of incomplete, poorly tested, and incorrect changes introduced into your codebase. They are a critical part of every effective development process to ensure that your solution matches your original intention, but can be time consuming. A good pull request description prepares the reviewer for what they’ll find during the code review. They can tell the reviewer where to focus their attention by highlighting areas that you’re not sure about as well as areas that you’re proud of. This makes it easier to process PRs during code reviews and quickly provide meaningful feedback.

6. Descriptions are a great source of knowledge

Most changes require some kind of research. You might have found a solution on StackOverflow, a blog post, or in the documentation. To get the answer, you would have to read a bunch of sources, match an approach to your problem, attempt the solution, and observe the result. This process is enlightening but also time-consuming. It’s important therefore to document your thinking in case someone else has to recreate it, but also because everyone can learn from it.

When you’re writing a PR description, all of this learning is fresh in your mind. It’s the best time to synthesize what you learned and share it with your team without the overhead of writing a blog post. Including lessons learned in your PR description will allow you to better remember the information that you learned and highlight what you learn for others on your team to benefit.

7. Descriptions make you invaluable

Most of our days as developers are spent finding solutions to problems. A PR description is equivalent to providing a great unboxing experience for your solution. If you spent a week working on a problem, spending an hour to write a detailed Pull Request description will make your solution look that much better because it’ll allow your teammates to understand why this problem was complicated and required a week of work.

Open source communities have a reputation for having great source code. It’s not a coincidence that they tend to have some of the strictest requirements dictating the quality of PR descriptions. Enterprise teams can learn a lot from the best practices of how open source communities write great software, starting with pull requests.

Requiring a good PR description is a small investment that pays big dividends. Give it a shot for a few months and encourage the same of your fellow developers. You might find yourself never going back, and emerge as a stronger and more valuable team member as a result.

The Triple Threat to Testing — Part 1: Speed

Mon, 06 Apr 2020 05:00:00 GMT

Writing tests is like paying taxes. And much like how attitudes vary towards paying taxes, attitudes diverge widely in development circles about writing tests. Some people dread paying their taxes; much in the same way as some developers would rather cut off their arm than spend time testing an app. Other people willingly pay their taxes because of the benefits they confer, like drivable roads, health care and national defense. In similar fashion there are those developers that look forward to testing because of the protection in gives against regressions and the confidence it inspires when making changes. You'll even find people who argue that there’s something inherently good in paying taxes and writing tests—a moral obligation if you will.

But just like people debate the merits of taxes, developers argue over the value of tests. Some claim that they don’t need tests while others think they are so important you should write them before anything else. There’s no consensus in part because tests vary widely across technologies, projects, teams, and organizations. One thing is for certain, however; just like taxes take money out of your pocket, testing imposes a variety of costs on your organization and your development team.

We’re not writing to convince you about the value of testing any more than we’re here to convince you to pay your taxes. But I think we can all agree, that if you are going to pay taxes, then you should expect the best governance in return. So when it comes to testing, our experience has show us that if you are going to invest in tests, there are three essential make-or-break factors that will dramatically affect how much benefit they will bring in exchange for how much burden they pose.

The first factor is the most obvious: speed. Slow tests are literally a drag on teams and organizations.

Why Speed Matters

In the stock market, information services offer different pricing schemes around speed. You can wait and get information for free in a minute or pay to get data within seconds. But if you want information with minimum latency the price tag goes through the roof. Why? The infrastructure to make this possible is more expensive and the value of knowing something before others is very high.

Speed gives brokers a competitive advantage, and the same is true when it comes to testing. How rapidly a test can diagnose issues in your application can make all the difference. Because tests determine the minimum rhythm at which developers can discuss the results of their code, the speed of a test suite scales up to affect the entire organization. A Pull Request whose test suite takes 1 hour to run sets a plodding pace not just for the development team but product and design teams as well.

What’s worse is that when tests don’t deliver results fast enough, a developer working on a feature loses the thread of the problem the test was intended to help solve. This dynamic quickly leads to frustration and causes developers—who are inherently problem-solvers—to find work-arounds that essentially avoid testing altogether. Whatever you think about testing, we can all agree that a test suite should be fast enough not to break the context under which it’s being run. An unused test suite isn’t a test suite at all.

With this level of dysfunction you’d think that improving test speed would be at the top of everyone’s to-do list. Yet if you ask around it seems like we have just accepted that tests are slow and have learned to adapt to their plodding pace. Some teams decide to run only a few tests on every PR through CI, reserving the whole suite for ‘important’ builds. Other teams choose from time to time to not even bother running tests locally.

These reactions are typical and even make sense: waiting for a test suite to run is boring and non-productive, like standing in line at the Department of Motor Vehicles. What’s worse is that you have to go to the back of the line time and time again when the scripts you are running produce results that are flaky. It’s depressing when the development process—which at its core is an essentially creative process—has come to feel bureaucratic as a result of testing.

What’s slowing down the test suite?

We have all experienced slow test suites. Maybe you’re even reading this article while your PR is running tests. Understanding why tests take the time they do allows developers to pre-emptively act and make them more effective and useful while also making them faster. In fact here are things developers can do right now both application-wise and with regard to test strategies to improve the speed of tests.

"Slow" Front-end Apps

User interfaces are supposed to be used by humans, but when a machine uses them they reveal their underlying performance issues. A person furiously tapping your app buttons as fast as they can will not match the speed of a machine. Not even developers themselves – who know exactly where each button is located – can assert the state of the app as fast as a script. That’s why slow loading and reacting UI elements are invisible to us most of the time.

When you run a test the machine will always be limited by the speed of the application. On every simulated interaction and on every assertion, the test runner waits: it waits for code to be loaded and parsed, for elements to appear, for APIs to load, or for interactions to happen. Repetition scales up the natural latency of an app until it becomes noticeable.

Right now you can do the following in order to streamline front-end apps:

Check the Time to Interactive of your application You may be loading too many things on the entry point of your app: user profile, dynamic navigation segments, analytics scripts, and more. Consider if there’s something you shouldn’t be using within the test context (like pixel scripts for marketing).
Disable non-test-related latency: for example, unless you’re testing the animation itself, you can consider disabling animations in the testing context. Animations just mean waiting time for the test runner in most cases.
Evaluate the overall performance of your app: this seems obvious, but sometimes it pays to stop and contemplate what's right in front of you. If the test are painfully slow, then perhaps your app is slow, and an investment in making it more peppy will yield dividends for both your developers and your users.

Slow Build times

The first step of a test suite usually involves generating a fresh build of your app in the context that you choose. Although this is a time cost that the test suite generally only has to pay once, it is still a major factor in how soon developers can get test results. A fresh build can take up to several minutes – the equivalent of eons in computer time.

Most applications today use a bundler like webpack to take the application code (and its libraries) and transpile it, minimize it, optimize it, modularize it, and do whatever other cool process the development team has chosen. As the application grows, the build time may grow non-linearly if they are not mindful of how they create the bundle itself. And if running a test requires spinning off a Kubernetes instance or installing a full-blown database, you can expect even longer wait times before you get any results from your test suite.

Right now you can do the following in order to speed up build times:

Budget for performance improvements: build times tend to be a boiling problem: as features are constantly added, the time increases without anybody noticing. Performance improvements are typically triggered only after somebody complains about the tools being too slow. We recommend having a performance budget to proactively improve the build times; otherwise you might find yourself returning to re-write the build script again and again.
Evaluate your test environment: check with your team if you can simulate parts of the test environment (for instance, use a tool like Mirage.js instead of using a real database).

Not specific enough entry points

Web applications are attractive because you don’t need to install anything to use them: you just hit a URL and you’re ready to do what you want. The URL is the entry point for a web application. Most people access the app through the root / and then navigate their way around. Advanced users may even know the URLs they care about and go straight to it (/p/{my-crush} or even /p/{my-crush}/photos). Using more specific entry points saves time of navigation to users, and can save time when testing.

Let’s imagine a developer wants to test a confirmation modal that shows up after a user attempts to change their Address in the user Profile, under the Personal Details tab. If, you have to manually follow all the interactions from the homepage to check if that the modal appears (as in the code below) your tests will take longer.

visit('/');
click('[navigation-login-button'];
waitFor('[login-form'];
fillIn('[login-username]', 'username');
fillIn('[login-password]', 'password');
click('[login-submit']);
waitFor('[logged-in-user]');
click('[navigation-my-profile]'];
waitFor('[profile-navigation]');
click('[profile-navigation-personal-details]');
waitFor('[personal-details-form]');
// Now we are finally ready to test what we care about
fillIn('[personal-details-form-address]');
click('[personal-details-form-submit]');
assert.ok('[personal-details-confirm-modal]');

That’s a lot of actions for one assertion – 10 that are unrelated to the test, and several of them involve waiting. If there are four scenarios you want to test about the behavior of that model, that’s 40 time-consuming and unnecessary actions.

Instead you can make use of your router and set up an authentication mechanism that allows you to easily set the application in an authenticated state. Then your test would be simplified and much faster:

simulateAuth('user');
visit('/profile/personal-details');
fillIn('[personal-details-form-address]');
click('[personal-details-form-submit]');
assert.ok('[personal-details-confirm-modal]');

Right now you can do the following in order to improve entry points:

Evaluate your routing strategy: in many cases using more routes with specific contents is not only good for tests but also helps deliver a better user experience.
Simulate authentication: discuss with your team the best strategy to simulate authentication in your application, from using a framework specific tool to manipulating local storage directly (while not forgetting to take security considerations into account).

Waiting too much, waiting too little

Asserting the state of a web application is like playing the street basketball game 21. The first team to score exactly 21 points wins; but if they go over their score is reduced by 11. Assertions are similar in that you must make them at a specific moment in time: if you don’t hurry up you lose precious time, but if you miss the mark you’re stuck with more waiting and have to try again.

Web applications perform a lot of tasks asynchronously: loading files, fetching images, calling APIs and recording analytics to name a few. That’s why waiting is such a tricky part of testing user interfaces. There’s no exact fixed time intervals in the way elements are rendered because they depend on many other elements and which client is being used. There’s also external dependencies like server requests that may take variable amounts of time.

Sometimes developers can set tests to wait too much. For example, to make sure an element was rendered in the page we can make our test wait for a fixed time interval like wait(‘500ms’). But if that element appears in 25ms, we would be wasting 475ms which add up rapidly through the whole test suite. Other times developers can waste time setting the test to wait on the wrong element, which could imply waiting longer than the minimum required for what we care about for a particular assertion.

Of course other times developers wait for too little time, using a fixed time interval that simply wasn’t long enough to render the element and causing an error when interacting with it. Depending on the tool we’re using, we can try again and re-run the waiting or get a test failure. In both cases we lose time because of inadequate waiting.

Right now you can do the following in order to find the sweet spot for waiting:

Check for fixed waiting times in your test suite: try to replace fixed waiting times for other heuristics that can be resolved earlier.
Discourage arbitrary waiting via tooling: you can set up linting rules to warn developers when they try to use fixed waiting functions.
Understand how your test runner handles waiting: you may have quick wins available by leveraging the waiting strategies your test runner is built for. For example, Cypress offers route aliases to help you wait precisely for API responses.

Resource-intensive testing scenarios

You can vary just how close to reality you test a web application in. For example, to provide data to the app, you can use an API deployed on a QA environment, or spin-off a local database seeded with data samples, or even pack mocks as part of the test. The closer to production that we emulate the testing scenario, the more resources – and time – will be needed.

The most significant advantage of getting a test environment closer to production is that you gain confidence that your tests are diagnosing your application realistically. However, running a test suite against real APIs represents a major challenge in terms of time and cohesiveness. The same can be said of other external dependencies (like Bluetooth connectivity) for your application.

Right now you can do the following in order to select the right testing scenario:

Assess the level of realness your team needs: perhaps your data needs are not too complex, and your application doesn’t handle critical operations, so mocks would suffice. If your business requires more sophisticated data management, you may need to invest in tools that can help you ensure your tests cover such cases.
Consider simulation: simulation is an intermediate point between mocks and real APIs that provides the velocity of mocks with the complexity of real backends. You can simulate database queries, as well as errors, using tools like Mirage.js.

Everybody feels better about paying taxes when they can see results. At Frontside we think everyone would feel better about tests if they could see their results faster. By addressing both application concerns (time to interactive, build time, routing granularity) and testing strategies (waiting precision, realness) developers right now can win back precious seconds, minutes, and even hours of time spent on slow tests.

We think it’s worth the effort to improve testing, as speedy test suites shorten feedback loops and add up to increase the organization’s velocity. And that maximizes the dividend you get for your investment in tests.

Up next...

Speed is critical to a healthy testing strategy, but it won't work without the other pillars of the testing triple threat. Next we focus on the second essential make-or-break factor in testing: reliability. Tests can be flaky, but there are strategies you can take now to detect, fix, and even prevent having non-deterministic tests in your application.

Keep on reading, The Triple Threat to Testing — Part 2: Reliability.

Case study — Facilitating the next generation of Bluetooth-connected apps

Taras Mankovski — Wed, 11 Dec 2019 05:00:00 GMT

Today Bluetooth is among the most widely adopted wireless communication technologies, and the popularity of Bluetooth enabled devices shows no signs of warning. With an anticipated annual growth rate of 8% through 2023 [1], it’s no surprise that more organizations than ever before are developing software to connect to and communicate over Bluetooth.

However, developing for Bluetooth can be cumbersome. The typical process begins with the frontend team being blocked until they receive the physical devices, and engineers having to wait to try out their work due to limited hardware. To show their progress, every month or two teams have to drop everything to invest at least a week’s time setting up a demo.

This rhythm makes course corrections from the VP of Product expensive, forcing the UX team to design some experiences as an after-thought. No less important is the emergence of a demo-driven culture that undermines everyone’s morale. And worst of all is the fact that it’s virtually impossible to have a fully automated test suite, dramatically increasing risks in the development pipeline.

When Resideo came to Frontside with these concerns, we partnered with Polidea—a development studio based in Warsaw, Poland—to implement a Bluetooth simulator. By creating virtual representations of Bluetooth peripherals, developers can keep working despite the unavailability of physical devices, designers can get early feedback of the application UX, and managers can get a solid sense of the progress of the project. Adopting this approach has the twin virtues of both easing developmental bottlenecks and enabling automated testing.

This case study provides an overview of the advantages of Bluetooth simulation and Frontside’s approach to open source collaboration with Polidea and Resideo.

Why use Bluetooth Simulation?

The idea of creating a Bluetooth simulator emerges from Frontside’s guiding philosophy of transparent development. It’s our belief that the impact of every change to the codebase can and should be tracked and be visible to both technical and non-technical stakeholders. We help organizations establish tooling and practices that enable them to get a real sense of their product as they build it.

That means that applications that connect to Bluetooth peripherals should follow transparent practices. Every pull request should be thoroughly tested, including the application’s connections to Bluetooth under varied circumstances and multiple configurations. And any stakeholder should be able to preview and experience every change in the codebase without particular setups or configs, just as a user would in production.

Relying on physical devices to achieve transparent development is not a viable option as they impose several restraints on teams. Their limited numbers force developers to share devices, limiting their capability to work concurrently. Testing the application with every supported device is a monumental task on its own. Moreover, visibility on the project is compromised because setting up demos demands significant effort.

But thanks to Bluetooth simulation, the VP of Product in San Francisco can test work delivered by the team in London and perform course corrections on the spot. The UX designer can adjust the connection drop-out experience from their home after reviewing the work done by a consultant in Stockholm while they were sleeping. No less important is the fact that everybody on the team rests easy knowing that thanks to automated testing the product works with every possible device configuration.

A win-win-win Open Source strategy

With Frontside’s approach to Open Source, companies, contributors, and the community all benefit. For this project, Frontside identified the need for a Bluetooth simulator as a key need for Resideo and saw the opportunity that creating one represented for the open source community. Polidea was commissioned to develop the Bluetooth simulator, Resideo provided financial support, and Frontside contributed architectural guidance to make the most out of the library.

Resideo, Polidea, and the Open Source community benefit from BLEmulator

Resideo Wins:

A library made by the best team for the job: by researching open source projects in the technologies relevant to Resideo, Frontside found Polidea, who had a proven track record of working with Bluetooth and experience in maintaining open source projects.
Access to highly-specialized experts on demand: corporations would typically be unable to hire smaller teams or experts to develop targeted solutions for precise pieces of the puzzle because of the associated transaction costs.
Reduced costs of ownership: now that the Bluetooth simulator is created, the maintenance costs and effort are shared by other contributors using the library.

Polidea Wins:

Financial support: Polidea already had plans for expanding its open source portfolio out of its own budget, but Frontside’s collaboration reduces this expense, enabling Polidea to focus on creating high-quality open source code.
Architectural insights: Frontside provided use-cases and architectural advice to make the Bluetooth simulator a tool that could be deployed independently of Polidea’s own libraries.

Community Wins:

A polished open source Bluetooth library: characterized by the rigor and quality of Frontside’s code and the extensive experience of Polidea, the community can now count on a more complete Bluetooth library in the Flutter space
Proved abstractions: Frontside and Polidea invested significant effort to create lean low-level abstractions that produce effective developer ergonomics for working with Bluetooth.

What’s Next

BLEmulator logo

Currently Polidea’s Bluetooth simulator BLEmulator only works with projects that have chosen FlutterBLE as the Bluetooth interactor. But Frontside aspires to maximize the value and impact of its software collaborations. That’s why BLEmulator is designed to be decoupled from FlutterBLE so the library may become independent and be more widely adopted. It features low-level abstractions using native implementations under the hood that enable API consistency with platforms like iOS and Android.

Learn more about BLEmulator from Polidea’s technical release and the project’s Github.

[1] Bluetooth SIG, Inc. (2019). “Bluetooh Market Update.” Available online at < https://3pl46c46ctx02p7rzdsvsg21-wpengine.netdna-ssl.com/wp-content/uploads/2018/04/2019-Bluetooth-Market-Update.pdf >

Build a Family Tree maker using React Hooks and Microstates

Taras Mankovski — Tue, 06 Nov 2018 05:00:00 GMT

Demo of Family Builder Component

In this tutorial, we will create a Family Tree marker component using React Hooks and Microstates. The Family Tree maker will allow the user to enter their name, then their parent’s names, their parent’s parent’s names and their parent’s parent’s parent’s names, as far back as they can remember.

To do this, we’ll follow the following steps:

Create a new project and install Microstates
Build recursive Family Tree maker component
Persist component state in LocalStorage
Optimize re-rendering of our component

Let’s get started.

Create a new project and install Microstates

Let’s create a new app using create-react-app. If you don’t have create-react-app installed, you can install it by following instructions on create-react-app website.

create-react-app family-tree-app
cd family-tree-app # go into created directory

Then add @microstates/react & microstates to your project.

npm install --save microstates @microstates/react

You should be able to start the server using npm start and see the React logo when you go to http://localhost:3000/.

Build recursive Family Tree builder component

Demo of Family Builder Component

Our component will allow the user to enter their name. When the name is entered, they’ll see an input field to enter names of their mother and father. When a parent’s name is entered, we’ll show input fields for parent’s parents. This will work recursively as deeply as the user has patience to enter.

State management is usually the most difficult part of building this kind of component. Luckily, Microstates will make this very easy. All we need to do is create a type that will allow us to store a name for a person, a mother and a father with their names. It needs to be recursive, just like our component. Microstates makes it very easy.

class Person {
  name = String; // name of the person 
  mother = Person; // mother is of type Person
  father = Person; // father is of type Person
}

Our type is complete. This is everything that we need to create a recursive data structure. Microstates will take care of proving us with transitions and handle the immutability of this recursive data structure. Now, we just need to write the component that will use our Person type and useType hook. Let’s begin by creating the state and passing it to our future FamilyTree component.

import useType from '@microstates/react';

function App() {
  let person = useType(Person);
  return <FamilyTree person={person} />;
}

We need to actually write our FamilyTree component. The component is going to show an input field for the name. When the name is not empty the component will render a FamilyTree component for the mother and the father.

function FamilyTree({ person )} {
	return (
		<>
        <input
          value={person.name.state}
          onChange={e => person.name.set(e.target.value)}
        />
        {person.name.state !== "" && (
			<>
				Father: <FamilyTree person={person.father} />
				<br>
				Mother: <FamilyTree person={person.mother} />
			</>
        )}
      </>
	)
}

That should do it. Look at the onChange handler of the input. Notice the person.name.set() transition. It’s similar to a reducer that changes the name of the node, but Microstates handles immutably updating the value and causing our component to update via our useType hook. Let’s use another hook to persist our input in LocalStorage.

Persist component state in localStorage

Every transition creates a new value that we can serialize and later use to restore the state. We can use this to persist the state in LocalStorage. When the user refreshes their page, the family tree will be restored from LocalStorage. To do this, we’ll need to change a few things.

Restore state from localStorage
Create the microstate with restored state
Persist the state in localStorage

Restore state from localStorage

When the component is being rendered, we need to grab a key from localStorage and parse it with JSON.parse. This part is not unique to React Hooks or Microstates.

const initial = JSON.parse(localStorage.getItem("family-tree") || "{}");

Notice the string wrapping an object ”{}" That’s an empty object that was serialized. I’m using this to ensure that there will be a value even when localStorage doesn’t have anything. This will happen on first rendering the component.

Create the microstate with restored state

Once we have the initial state from localStorage, we can pass it to our App component and create the microstate with the passed in value.

const initial = JSON.parse(localStorage.getItem("family-tree") || "{}");

function App({ initial }) {
  let person = useType(Person, initial);

  return <FamilyTree person={person} />;
}

ReactDOM.render(<App initial={initial} />, document.getElementById("root"));

This will allow us to deserialize the value, but what about saving the value into localStorage? We’ll do that next, it’ll require that we use another hook.

Persist the state in localStorage

We want the state of the family tree to be saved in localStorage. To do this, we need to extract serializable state from the microstate. We can use valueOf function to get a value that we can store in localStorage.

import { valueOf } from "microstates";

function App({ initial }) {
  let person = useType(Person, initial);

  let value = valueOf(person);

  return <FamilyTree person={person} />;
}

Once we have the value from the microstate, we can use useEffect hook to queue up the localStorage.setItem operation. useEffect hook is executed after the render is complete, which allows you to perform potentially slow operation without slowing down rendering. It also allows you to prevent unnecessary invocations of this effect by declaring the value as a dependant value. Here is what that looks like,

import React, { useEffect } from "react";
import { valueOf } from "microstates";

function App({ initial }) {
  let person = useType(Person, initial);

  let value = valueOf(person);

  useEffect(
    () => {
      let serialized = JSON.stringify(value);
      localStorage.setItem("family-tree", serialized);
    },
    [value]
  );

  return <FamilyTree person={person} />;
}

In this section, we added the ability to store user’s input in localStorage to allow us to restore their input when the user returns to the page. We used valueOf to extract value from the microstate and useEffect hook to ensure that calling our persist operation does not block rendering. In the last part of this tutorial, we’ll optimize the component to ensure that it doesn’t re-renders parts of the tree where state did not change.

Optimize re-rendering of our component

It’s not unusual for performance improvements to be left until the end of the project, but unusual for performance tweaking to be easy. Fortunately, Microstates was designed in a way that makes optimizing performance with React easy. In this section, I’ll show you how to optimize the component to ensure that only parts that changed actually re-render.

The key to optimizing React rendering is to ensure that only components that changed get re-rendered. React Devtools has a feature called “Highlight Updates” that makes it easy to see which components get re-rendered when you interact with the application.

Demo of Family Builder Component before optimization

When “Highlight Updates” is turned on, the DevTools will highlight areas of the component tree that are being updated. If you look at the example above, you can see that every keystroke causes every component to re-render. This is a lot of unnecessary re-renders.

What we need to do is to only re-render children when the state of children has changed. We can use the fact that Microstates are immutable as a guarantee that microstates will only change their references when their value has changed. If they value did not change, then the same microstate will be reused.

We can combine this reference guarantee with the useMemo hook to memoize components based on the microstate that they consume. This will ensure that memoization will be invalidated when the dependant microstate changes. Let’s modify our component to memoize parent’s components based on their microstates.

function FamilyTree({ person }) {
  const father = useMemo(() => <FamilyTree person={person.father} />, [
    person.father
  ]);

  const mother = useMemo(() => <FamilyTree person={person.mother} />, [
    person.mother
  ]);

  return (
    <>
      <input
        value={person.name.state}
        onChange={e => person.name.set(e.target.value)}
      />
      {person.name.state && (
        <ul>
          <li>Father: {father}</li>
          <li>Mother: {mother}</li>
        </ul>
      )}
    </>
  );
}

After we make this change, we can check the result and see if we made an improvement. Now when you edit an input field, it’s only changing components above the component that you edited because their microstates were changed.

Demo of Family Builder Components after optimizations

It’s worth noting that we only change the input in one component but all of the parent components are marked as changing. This is because when a nested microstate changes, the parents of that microstate have to be re-created as per rules of immutability.

Conclusion

In this tutorial, use used useType hoook to create a recursive component using Microstates, stored our state in localStorage and optimize re-rendering of our components.

If you thought that this was easy, then consider how much we were able to do with Microsates and React in 80 lines of code. You can see the final result in GitHub Repo and CodeSandbox.

If you thought this was difficult, please file an issue and describe what you found difficult.

Regardless, tweet me @tarasm and let me know what you think about Microstates and useType hook. If you’re looking for help building large applications, consider hiring Frontside to help you build your app.

Four reasons to use append() instead of Object.assign() and Object spread

Charles Lowell — Tue, 18 Sep 2018 21:00:00 GMT

Warning! This post was manufactured in a facility that also processes Monoids and may contain traces of functional programming.

I'd like to share with you what is easily one of my top 5 favorite discoveries in programing during the last year: The append operation for JavaScript Objects. Before you go looking in the API docs: no, this is not part of the official ECMA Script specification. It's a pattern that I found on the internet and implemented in funcadelic.js. You can use my implementation, or you can write your own, but either way, once you experience how much better append tastes than its analogues in vanilla JavaScript, you'll have a really hard time going back.

The TL;DR is that you can use append anywhere that you would use Object.assign or the object spread operator except it's more awesome. How? Well I'm glad you asked.

`append` is immutable.

It never changes any of its arguments, only ever creates new objects that are derived from the objects that you give it.

let ingredients = append({ eggs: 2}, { mushrooms: 10 })
//=> { eggs: 2, mushrooms: 10 }

I'll not try to convince you of the benefits of immutability here, only say that as programmers, we truck in that most precious of resources: information. Why willingly destroy it unless absolutely necessary?

When you use append instead of Object.assign, you'll never have to pay the overhead of that decision, or worse, accidentally mutate some object you never intended to.

`append` preserves type.

Every JavaScript object has a runtime type. This lets us define custom properties and operations on it. Unfortunately, if we're making an immutable transformation using Object.assign or Object spread, that type information is erased and no matter what we started out with we revert to a plain old JavaScript object.

class Person {
  constructor(firstName, lastName) {
    this.firstName = firstName;
    this.lastName = lastName;
  }

  get fullName() {
    return `${firstName} ${lastName}`;
  }
}

let misspelled = new Person("Tony", "Stork");

let corrected = {...misspelled, { lastName: "Stark" } }
//=> Object { firstName: "Tony", lastName: "Stark" }

corrected.fullName
//=> undefined

Notice how the fact that we're work working with a Person object is just thrown into the dumpster? By contrast, append always preserves type of object that we were working with before.

let misspelled = new Person("Tony", "Stork");
let corrected = append(misspelled, { lastName: "Stark" });
//=> Person { firstName: "Tony", lastName: "Stark" }
corrected.fullName //=> "Tony Stark"

I mean you could throw away the type by default, but why?

`append` is lazy

JavaScript has a really nice feature called getters or "computed properties" that lets you defer evaluating a property's value until it's actually needed. This can be a great way to avoid computing a property if the dependencies required won't be available until later, or you'd just rather not pay the cost of evaluation until you absolutely need to.

Unfortunately, whatever your reasons, both Object.assign and Object spread will interfere with them because they eagerly evaluate each property of the objects being copied over into the result. For example, let's look at these two objects that have computed properties:

let left = {
  get thousand() {
    console.log('calculating 1000 in a really expensive way');
    let i = 0;
    for (; i < 1000; i++);
    return i;
  }
}
let right = {
  get hundred() {
    console.log('calculating 100 in a really expensive way');
    let i = 0;
    for (; i < 100; i++);
    return i;
  }
}

Let's see what happens when we combine them with Object spread:

let both = {...left, ...right};
// [LOG] calculating 1000 in a really expensive way
// [LOG] calculating 100 in a really expensive way

Or how about Object.assign?

let both = Object.assign({}, left, right));
// [LOG] calculating 1000 in a really expensive way
// [LOG] calculating 100 in a really expensive way

But we made those properties lazy for a reason! Just because I'm deriving a new object based off of those properties, doesn't mean I should have to evaluate them! Well with append you don't. It will follow the guidelines laid down by each object's original creator and keep normal properties normal, and computed properties computed.

let both = append(left, right);
// nothing printed, because nothing evaluated!
// only when we access the properties does the computation run.

both.thousand //=> 1000
// [LOG] calculating 1000 in a really expensive way

both.hundred //=> 100
// [LOG] calculatingc 100 in a really expensive way

`append` is a universal interface.

Remember when I said that I found append on the internet? That was true, but it also wasn't the full story. What I actually found on the internet was something far more powerful: the concept of a semigroup. A semigroup is (roughly) a type; any two of whose values can be combined with each other to produce a new value of the same type. For example, JavaScript Array is a semigroup. I can combine or "append" any two Array objects together to produce a single new Array:

[1,2] + [3, 4] //=> [1, 2, 3, 4]

JavaScript String is a semigroup too!

'Hello' + ' World!' //=> Hello World

This got me to thinking: what would a semigroup for object look like? What would it mean to take two Objects and combine them to produce another Object, and the append() function that I've been talking about was the inevitable consequence:

Immutable because this is functional programming, and that's really just a given.
Type-preserving: because the append operation of Semigroup must return the same type as its arguments.
Lazy: This isn't strictly a requriment, but as long as we're building an amped up version of Object.assign we might as well respect object property semantics, no?

This is all to say that using append normalizes this operation to a shared understanding of what a Semigroup is. And in fact, in funcadelic.js, the append operation will work on all these types.

append([1,2], [3, 4]) //=> [1, 2, 3, 4]
append('Hello', ' World!') //=> 'Hello World!'
append({ eggs: 2 }, { bacon: 2 }); //=> { eggs: 2, bacon: 2 }

One concept, one function, lots and lots of different scenarios. And that's why I started using append and haven't looked back since.

Kubernetes for the Kubernewbie - The Journey

Elrick Ryan — Thu, 09 Aug 2018 21:00:00 GMT

‘It’s not how you start it’s how you finish’, is a phrase you may hear a coach utter to his team but it is surprisingly applicable in the Tech industry as well. All of us were newbies at some point in time. We’ve all faced challenges while trying to learn something new. However, reaching the end of the road to proficiency requires you to push through those tough times and I am here to let you know that it is possible even when entering Kubernetes (k8s) as a Kubernewbie. My background prior this undertaking was Design and Development (DevSigner), and not Ops. However, by the end of this journey I made the transition and gained a new moniker, DevOpsSigner.

Initially, this blog post was going to serve as a comprehensive intro to Kubernetes but, after some thought (and a few rewrites), I realized that this would not be possible in a single post. It was better to approach this post from the angle of what this is not meant to be. This post is not a comprehensive introduction to Kubernetes, nor is it a tutorial or even a guide. It is more akin to a testimonial, a recap of our experience coming into Kubernetes and how we went from Kubernewbies to a fully deployed production application in a Kubernetes cluster in a few weeks.

With no prior experience with Kubernetes or Devops it made the process of learning and ramping up even more challenging. When entering a new domain, like Devops, there is a load of domain specific knowledge (DSK) you do not possess that others have acquired over time. That DSK is what you use to evaluate, explore, build solutions and use technologies in that domain. Without it you could go down a path of reinventing the wheel. Every domain has very complex problems that people are trying to solve, have solved or are improving on. Fortunately for us and you that solution maybe primed and applicable to the need / issue / requirement that you have. Kubernetes was that soltion for us.

Kubernetes is a solution to a complex problem in its domain and by using Kubernetes you cut a chunk of complexity in this initial DSK acquisition and can start your Kubernewbie and Devops journey, ‘standing on the shoulders of giants’. Kubernetes gifts you a system with a set of primitives, an API, and a set of hooks to build and focus on things unique to your needs. Joe Beda said it best in that ‘Kubernetes gives you a Platform to make a Platform’. We will talk about how we used this ‘Platform Builder’ (Kubernetes) if you will, later, but first let us define what Kubernetes is in general.

What is Kubernetes?

Kubernetes.io introduces Kubernetes as “an open-source system for automating deployment, scaling, and management of containerized applications.” Kubernetes builds upon 15 years of experience of running production workloads at Google which was influenced by Google’s internal systems Borg & Omega. So what does Google know about production workloads you ask? Well around 2015, Google was spinning up approximately 2 billion containers per week. Yes, you read that correctly, 2 billion per week. I don’t know about you, but I have not seen 2 billion of anything in even a year.

So, it is fair to say that Google has a tremendous amount of experience utilizing containers in production. Fortunately, some brave souls started to open-source that expertise. Harnessing that experience, then Google employees Joe Beda, Brendan Burns andCraig McLuckie took on the challenge. Brendan Burns started the initial prototypes which, combined with work from Joe Beda and Craig McLuckie, resulted in the first commit to the Kubernetes repo in 2014.

Kubernetes was created to bring the idea of dynamic, container-centric, managed, scheduled-cluster thinking outside of Google. With the exponential growth and usage of containers globally, Kubernetes strives to give everyone a production level solution fashioned from Google itself. There is no denying Google’s experience with containers and containerized applications. But what is a container and what does a containerized application mean in this context?

What is a containerized application?

For now, let’s say that a containerized application is an application that is within a container. A container, by definition from Webster’s Dictionary, is “an object that can be used to hold or transport something.” In relation to technology, that “something” is everything that a piece of software needs to run in a particular environment, such as application code, utilities, configuration and dependencies. So a proposed Tech definition is: a container is an abstraction: an Operating-system-level-virtualization that allows you to run an application and its dependencies in resource-isolated processes.

Diving even deeper into this tech definition, an Operating-system-level virtualization, also known as containerization (containerized application), refers to an operating system feature in which the kernel allows the existence of multiple resource-isolated user-space instances. Those instances are the containers that and share one Host Operating system across them. Since the containers encapsulate everything an application needs to run this makes moving and running the software in different environments possible. So with all these definitions in hand, my final description of a container would be: a Container is a isolated-resource that is used to transport everything an application needs to run.

That was an intense explanation of what a container is, but container technology is a fundamental technology and a catalyst for the building of Kubernetes. Kelsey Hightower said, “People will soon learn that containers only solve the software packaging and distribution problem. Containers don't manage anything; they need to be managed”. Kubernetes presents a management solution for containers. Lets now assume you have some containers built and ready for use, you can have anywhere from one to one-thousand of them to deploy -- and you’ll need a way to deploy, manage and even scale the system using these containers.

That was the position we found ourselves in with a new project that we were hired to develop. We were hired to build a React application and a backend application. The backend application was developed using Ruby, which would be a Microservice / module living in an overall system of modules, powered by an API-Gateway called Okapi. We needed our own production and sandbox environments that were fully functional versions of the overall Microservice Architecture. Having this would allow us rapid development and iteration on our module, a place to demo all the new features we would be implementing, and a sandbox location for experimentation. Since our module was going to be part of a larger system made up of multiple modules/containers we needed a deployment and management solution. Enter Kubernetes.

Why did we need and settle on Kubernetes?

We researched a few options for a container management solution prior to settling on Kubernetes. Many of which are capable of being a solution to container management but ultimately we went with Kubernetes because of a few things:

Documentation. The Kubernetes documentation is extensive.
Self Healing. An essential feature of Kubernetes is ‘self-healing’, the system will always try to keep your cluster of deployed pods ( which hold containers) running at the desired scale you define. Yes, you can ‘set it and forget it’ in Tech.
Declarative. Kubernetes is a declarative system and uses files written in YAML or JSON to define how you want the various parts of the system to look. There are more advanced solutions, rather than hand rolling and editing multiple YAML files, but these are used at the basic level.
Community. There were several members from the Kubernetes community who were extremely helpful and receptive to answering questions as we ramped up our experience.

Managed or unmanaged hosting?

Now that we’d settled on using Kubernetes, we needed to decide on where to host our Kubernetes clusters, and whether we wanted to go with a managed or unmanaged Kubernetes hosting solution. A managed solution has Provisioning, Security, Load Balancing, Upgrading, and Monitoring that is handled by them. An unmanaged solution will require you to do more of the leg work to setup all the previously mentioned points. However, not all managed Kubernetes hosting solutions are the same, so we researched a few. One that was pretty awesome was Platform9, which is a good turn key managed solution, but we ended up going with Google Kubernetes Engine (previously Google Container Engine).

Google Kubernetes Engine (GKE) is a managed solution that abstracts away the aforementioned Provisioning, Security, Load Balancing, Upgrading, and Monitoring of running a Kubernetes cluster, but not so much that you cannot configure (and even swap) an alternate technology for one of those parts. However, staying as close as you can in the GKE system and using all the features within GKE will make your life much easier. Now that we’ve decided on using Kubernetes for orchestration, and GKE as the managed solution for hosting the cluster, let’s Ship it. Well, not so fast.

Tools within the Kubernetes ecosystem we use and building our platform.

Before we actually shipped to GKE, we flexed our Kubernetes muscles and experimented with Kubernetes using Minikube. The Minikube readme says that,

“Minikube is a tool that makes it easy to run Kubernetes locally. Minikube runs

a single-node Kubernetes cluster inside a VM on your laptop for users looking to try out Kubernetes or develop with it day-to-day,”.

We used Minikube to initially test standing up and deploying all the modules. Minikube allowed us to get familiar with certain parts of Kubernetes, known as Resource Types such as Pods, Services, Replica Sets, Deployments, Config Maps etc and go through some of the examples in the Kubernetes documentation and around the web.

During this k8s exploration period using MiniKube we became very familiar with using another tool called 'kubectl'. kubectl is a command-line interface (CLI) for executing commands against a running Kubernetes cluster in this case the running cluster was the single-node Minikube instance but could be any running k8s cluster. kubectl has a laundry list of commands but our initial use was to ‘apply’ our YAML files and as time progress we started to use more of the advanced commands of kubectl. One resource that I wish I had during this exploration phase was a book called ‘Kubernetes Up and Running’ which I read later but would suggest it to anyone that is new to k8s. Despite not having the book at the time this step was crucial to gaining a basic understanding of Kubernetes and standing up a cluster before shipping the full ancolata to GKE, which we did soon thereafter.

We popped some champagne and celebrated the win of successfully standing up a Kubernetes cluster with a fully functional API-Gateway and modules (including our module). Our clusters were running great, but after a few weeks and having to swap out and redeploy parts of it (and on occasion having to redeploy the entire cluster), we needed to find a solution to automate the process of tearing down and standing up our cluster. Again, we did some market research, but we couldn’t find anything that would fit our needs at the time; so we ended up building Okubi, as mentioned earlier using the ‘Platform to build a Platform’. Okubi gives us a command line interface for tearing down and standing up our cluster.

We use Okubi for things like:

Generating module YAML files from a template.
Changing configuration values foreach environment from a configuration file.
Setting priority level to modules.This will allow explicit ordering of when certain modules are deployed.
Order the deployment of the certain parts of the system to the cluster.
Run execution steps during the standup specific to this infrastructure.

Okubi is great, but it took hitting the pain points of doing all this manually before we found a solution that worked for us and built an abstraction. Definitely have to give a big shout out to Joe LaSala for all his hardwork in Okubi. We have some enhancements that we will like to introduce within Okubi but currently it is serving us well with cluster deployment. While having to redeploy modules in the cluster or redeploy the cluster we often found ourselves debugging containers.

Debugging Containers

To debug containers one method that we use was to tail the logs of the container. Which essentially watches the logs and prints them to the terminal. Though you can run a command in kubectl to tail the log of a container. We started to use Stern because of some added features. Stern simply describes itself as Multi pod container log tailing for Kubernetes. This was a feature that we used along with tailing multiple containers within a Kubernetes “Pod”. It maybe possible to do the same with kubectl but do not quote me. However, using Stern made this task fairly simple with easy to remember CLI commands.

Tailing logs is very useful but then we wondered if we could somehow interact with the code while it was running in the cluster. Then we found Telepresence. Telepresence is an open source tool that lets you run a single service locally, while connecting that service to a remote Kubernetes cluster. So, by using Telepresence we were able to change the reality in your running cluster and tests code locally as though our local code was running in the cluster. In addition, we were able to find usage for Telepresence during active feature development. Yes, this all sounds quite magical and it did to me as first so let me try to explain how we used it.

First, use case was when we saw some buggy behavior and wanted to debug that code. We would then switch to our sandbox cluster context utilizing another tool called Kubectx. Kubectx says it is a way to switch between clusters and namespaces in kubectl and that is exactly right. Since our sandbox mirrors our production cluster we would then Telepresence into the sandbox cluster and then set debuggers within our local in order to step through the code during execution. Once we found the offending code that was causing the bug we would then create a PR with the fix.

Second use-case as I mentioned earlier was the usage of Telepresence during active feature development. This is one usage that came in handy for me several times. Let’s say I was task with developing a feature in the module. That new feature would then be consumed by the React frontend that I aforementioned React application that we had underdevelopment as well. Now, call me crazy but even though I have all these awesome testing tools at my disposal and full test coverage around my code because we do not ship code without tests. I still want to test it live. Hooray for Telepresence.

I would follow the same steps of making sure I was in the correct sandbox context and then Telepresence in that cluster. In addition, I would then spin up the React application locally and make sure it was making it calls to the sandbox URL. This would give me a live React, a Telepresenced Ruby Module and a fully running Kubernetes cluster. I could manually test out what I was developing and perform a full end to end test before I pushed to out a PR. I did not use this all the time but on the occasions when I had to reach for this technique it was worth its weight in Gold.

In closing

Well, that is all that I have for this post. Honestly, I could keep writing but this is supposed to be a blogPost and not a blogBook. So, I will keep the additional topics for another blogPost. Overall, I would say that the journey from a Kubernewbie was very challenging and extremely rewarding. You have to push your limits and go into uncharted territory in order to grow. Have faith in your abilities, ask questions and don’t get discouraged when you can’t figure something out. There is still much to learn but that is nature of the Beast which we call technology. If you have any question you can always hit me on twitter at @elrickvm or send an email. Be blessed and happy coding!

Photo by chuttersnap on Unsplash.

JSConf 2018 Recap

Robert DeLuca — Sun, 05 Aug 2018 00:00:00 GMT

JSConfUS has officially made a come back after a three-year hiatus. I’d like to take this moment to thank Chris & Laura Williams for all of the work that they put into building such an awesome & welcoming community. I’d also like to thank the new organizers that brought JSConfUS back! This year was exceptional, as you’d expect from a JSConf!

I had the pleasure to attend JSConf 2015 along with this most recent JSConf. This year JSConfUS was held in sunny San Diego. I prefer the resort in FL simply because there was direct beach access. Let’s be honest, that’s a total nitpick. 😆

One of the new twists to JSConf this year was the addition of a first come, first speak format. This allowed anyone to sign up and give a talk in Track B, which is exactly what I did! I wasn’t selected to be apart of the main track but that didn’t stop me from giving my talk on BigTest. I love this idea so much. There were a ton of interesting talks that came from Track B & it allowed some to speak for the first time.

My favorite seven talks

It would take me forever to recap each talk, so I’m going to narrow the field down to my favorite seven. Turns out when all the talks are good it makes it really hard to pick your favorite seven, but I’ll try anyway!

I See: Overcoming Challenges In Software Development

By Kevin O’Neil

In this talk, Kevin covers all of the challenges he’s overcame to be a front-end developer with a visual impairment. He covered a range of challenges like finding a job in a city that had good public transit since most jobs required him to be in office when he can’t drive. This is ultimately why Kevin, a San Diego native, ended up in Tokyo!

He also covered the challenges he faces when trying to do pixel perfect front-end development and the different browser extensions & tools he uses to get the job done, which I found fascinating. Tools like Atoms Telatype allowed him to pair with his coworkers without having to adjust a ton of settings.

I had the pleasure of sitting at the same table with Kevin before and after his talk. Once he was finished with his talk, one of the coolest things happened. A PM from the Firefox dev tools team came over and asked if there was anything they could do with their dev tools to help improve Kevin’s development experience. Which got me thinking… We put effort into making the experiences we’re building accessible but rarely give thought to accessible developer experiences.

ES6 In Practice

By Tim Doherty

I wasn’t sure what to expect when I sat down to watch this talk. My initial reaction was “I’ve been writing ES6 since 2015, what could I take away from this?”. I’m happy I did attend and didn’t judge purely from the title. Tim had a pretty awesome presenting style that I enjoyed. It kept me engaged and entertained as we took a tour through where ES6 came from, what actually made it into ES6, and what followed in ES2016-18.

This is a great refresher talk for anyone that has been using ES6 for a while as well as a great intro talk for those who are still looking to dip their toes into the new language features.

Please Don’t Mock Me

By Justin Searls

I had the chance to see this talk earlier in the year at Assert js, which blew me away. Justin is an excellent conference speaker with a ton of experience. This talk had something like 725 slides! I thought I had a ton at 250. This talk was slightly different from the one he gave at Assert. The main differences I can pick out are tweaked examples and a little more fine tuning on the message.

This talk was pretty damn awesome & a ton of information was thrown out there at a rapid pace. The main takeaway is: mocking in unit tests is supposed to help shape the design of your program rather than punching holes, in reality, to make it easier for you to test. It helps you figure out when and where the right time to mock in your unit tests.

This might have been the best talk from JSConf, but it’s so hard to declare that with all of the other amazing content. Lastly, I’m super impressed with the closed captioning service at JSConf. They were able to keep up with Justin and nailed it.

Adventures In Ethical Computing

By Myles Borins

This talk by Myles wasn’t technical but it was by far the most thought-provoking one I saw. He covered different ways bias works its way into the programs we write — whether we like it or not. We work our own biases in when we create. It’s up to you to recognize your bias and actively challenge them.

Myles gave a couple of solid examples to demonstrate these biases. My main takeaway from this talk is to always keep looking at and digging into the data you have. Is it fully formed? Where is the inherent bias in it? What can you do to challenge that and create a better data model?

This is one that you’ll want to watch once the video drops. 🔥

Exploring WebUSB And Its Exciting Potential

By Suz Hinton

As a part-time IOT nerd who is always looking for ways to automate my house, this talk from Suz was a lot of fun and super informative. Also big kudos to her for coming straight off a vacation and filling a speaker role that opened up in a weeks time. 👏🏼

Suz covered how the browser used to interact with physical devices connected via USB, what changed, and what’s coming down the pike. She had a live demo on stage, which is gutsy! But totally worth it because it was neat to see a full end-to-end demo. She plugged in an Arduino, allowed the browser to connect to the device, drew a message in the browser, and then sent it to the Arduino.

If you have any interest in IOT & controlling those devices visa USB through the browser, this talk is for you!

A Quest To Be A (Web)Master: How Do You Go From Simple Code To Building Great Web Experience?

By Mariko Kosaka

The best way I can sum up this amazing talk from Mariko is it covers all of the things Google engineers have been yelling at me for. But she did it in an awesomely informative way. I learned so much about how the browser works.

Her slides and diagrams explaining how the browser offloads different process to the GPU or other threads were SUPER helpful. If you’re struggling with getting silky smooth 60fps animations without page jank, I highly recommend checking this talk out. It made a murky picture pretty clear for me!

Discovering The True Identity Of @Horse_JS Using Machine Learning

By Burke Holland and Jasmine Greenway

This was probably the most fun talk at JSConf and they saved it for last! Burke & Jasmine detailed their experience trying to parse @horse_js’s tweets to figure out who @horse_js really is.

Turns out they weren’t really able to use machine learning too much to help them with this, but they did learn a lot about different data crunching techniques. There were a lot of hilarious jokes & puns sprinkled throughout the talk, which made it great to end the conf on.

They ended up narrowing horse down to 4 different possible people (who were never exposed, for good reasons!). One in ATX and three in NYC. Am I horse JS? 🤔 They never said who it was… the mystery will never be solved!

Conclusion

JSConfUS 2018 was amazing as all other JSConfs usually are. The new organizers did a great job making sure everyone felt welcomed and were super responsive to any questions I had. It was one of those conferences where the organization was done so well it fell away and was out of sight.

I tried my best to narrow my favorite talks down, but let’s be honest, all of the talks were amazing. If you have spare time when the videos come out I recommend watching all of them. They were diverse and full of super interesting information!

I can’t wait for JSConfUS 2019! My checkbook is waiting…

WCAG 2.1: What does it actually mean?

Robert DeLuca — Sat, 14 Jul 2018 16:00:00 GMT

Publication as a W3C Recommendation finalizes the development process and indicates that the W3C considers the updated guidelines ready for implementation on web content.

Last week WCAG 2.1 (Web Content Accessibility Guidelines) was recommended by the W3C, which means they have finished it and it’s ready for everyone to start implementing.

WCAG 2.0 was completed in 2008, back when touch devices were not as popular and tablets didn't exist yet. Since 2008 these devices have become the most prevalent way to access the web, but there were no success criteria that addressed accessibility for this input type. That is the main theme of WCAG 2.1: take into account touch devices. There are also a lot of new success criteria that address cognitive limitations too.

Within WCAG there are three levels to conform to: A, AA, and AAA. Level A is the lowest bar of conformance and level AAA is the highest. These WCAG levels cascade, so if your app conforms to WCAG AA, it also conforms to WCAG A. Most companies shoot for AA since AAA has success criteria that are a little hard to reach and not all content can conform to AAA.

In WCAG 2.1 there are:

5 new A success criteria
7 new AA success criteria
5 new AAA success criteria

Let's go over all 17 new success criteria to see what they are meant to solve and what it means for you. If you read the details for each of those success criteria some of them are straightforward and others aren’t. So I’m going to give my TL;DR version of the success criteria.

Level A

There are five new success criteria for level A, which is the lowest level of conformance you can achieve (aka the easiest). The new success criteria are:

2.1.4 Character Key Shortcuts

Assistive tech users have a lot of their own key combos for navigating and interaction. This success criterion is to ensure you don’t implement keyboard shortcuts that would prevent their functionality.

If you have any custom keyboard shortcuts implemented, one of the following need to be true:

The ability to turn off the shortcut
The ability to remap the shortcut to something different
The keyboard shortcut is only present if the component is focused

You can read the reasoning for it and many more details here.

2.5.1 Pointer Gestures

Since the last WCAG spec release there has been a lot of innovation on the web regarding touch devices. For example, with Google Maps you can use multi-touch gestures to pinch and zoom around the map.

This success criterions says if there are multi-touch or slide gestures present, there must be a single input method also available. To keep with the maps example, there should be + and - buttons available to provide non-pointer gesture interactions for this success criterion to pass.

If the gesture is “essential” you may not need to worry about this one. They define “essential” as:

If removed, would fundamentally change the information or functionality of the content, and information and functionality cannot be achieved in another way that would conform

You can read the reasoning for it and many more details here.

2.5.2 Pointer Cancellation

For interactions that can be done with a single pointer (clicks, single taps, long presses, etc.), you must provide a way to cancel it. One of the following must be true:

Don’t use the down event (touchstart or mousedown)
You have a way to undo the action
The up event cancels the down event (this is like clicking a link, moving your cursor, and releasing)

You can read the reasoning for it and many more details here.

2.5.3 Label in Name

Say you have an input with the visible label as “Name” and an aria-label as “Enter First Name”, this success criterion would fail. This is because speech input users navigate by saying the visible label. If the visible label doesn’t match the programmatic name (or accessible name) it will fail to navigate.

To drill it home with code, here’s two examples:

<!-- Bad -->
<label for="name">First Name</label>
<input type="text" id="name" aria-label="Enter your first name" />

<!-- Good -->
<label for="name">First Name</label>
<input type="text" id="name" />

You can read the reasoning for it and many more details here.

2.5.4: Motion Actuation

If you have an action that is triggered by motion, there must be a control on screen that performs the same action. For example, if you shake the device to undo text in an input, there should be a clear button available for single pointer devices to activate.

Another example would be images you interact with by tilting your phone around. There should be buttons available that pan / tilt the picture too.

You can read the reasoning for it and many more details here.

Level AA

There are seven new success criteria for level AA, which is the mid-level conformance you can achieve. The new success criteria are:

1.3.4 Orientation

Your site or app should work in either orientation: landscape and portrait. Unless it’s deemed to be “essential” to the function of your app:

Examples where a particular display orientation may be essential are a bank check, a piano application, slides for a projector or television, or virtual reality content where binary display orientation is not applicable.

You can read the reasoning for it and many more details here.

1.3.5 Identify Input Purpose

The inputs on your page should fill in metadata to help the user better understand the intention of the form inputs. You can do this by adding autocomplete attributes to your inputs. This is a full list of input purposes.

You can read the reasoning for it and many more details here.

1.4.10 Reflow

This might be my favorite new success criterion! It essentially means your site or app should be responsive. If you can’t make your site or app responsive an alternative could be providing a fixed 320pxwidth version.

The explainer document linked below goes into great detail about why this decision was made and is a great resource for understanding how to pass this success criterion. I recommend reading through this one!

You can read the reasoning for it and many more details here.

1.4.11 Non-Text Contrast

UI components and meaningful graphics must have a color contrast ratio of 3:1 to adjacent elements. This includes all states UI components can have (except the disabled state, which is confusing to me).

For example, a button’s hover, active, and focus states must all pass the 3:1 color contrast ratio. This does not apply to inactive elements (like a disabled button).

You can read the reasoning for it and many more details here.

1.4.12 Text Spacing

Content on your site or app must be able to expand to the following requirements without any loss of content or functionality:

Line height (line spacing) to at least 1.5 times the font size
Spacing following paragraphs to at least 2 times the font size
Letter spacing (tracking) to at least 0.12 times the font size
Word spacing to at least 0.16 times the font size

That does not mean that your site or app has to implement the above requirements as a baseline. But it should be able to scale to those requirements without losing functionality or readability.

This success criterion helps those who have low vision or dyslexia since they may override the default content styles to help reduce confusion while reading.

You can read the reasoning for it and many more details here.

1.4.13 Content on Hover or Focus

Any content that shows or hides from hover or focus needs to be:

Dismissable: a way to dismiss the content without moving the pointer or keyboard focus.
Hoverable: the shown content should be able to be hovered and not dismiss.
Persistent: the shown content remains visible until the user dismisses, focus trigger is removed, or the information is no longer valid.

The key takeaway here is if you have content that appears due to an action, the user must have control over that content being shown or hidden. It can’t disappear without a explicit dismiss action from the user.

You can read the reasoning for it and many more details here.

4.1.3 Status Messages

The main focus of this success criterion is for screen reader users. When there is a change in content that does not change focus you should provide a status message to the screen reader. This is commonly known as the announcer pattern. I’ve written an ember addon for this and there is a react component that does this too.

An example of this might be when an error on a form is displayed in a banner it should also announce that message to the screen reader through aria-live or role=“alert”.

Pro-tip: be careful when implementing this in your site or app. It’s very easy to get multiple announcers on the page that could announce over each other at any given time. I recommend having one announcer component that serves the entire application. That way there can only be one announcement at a given time.

You can read the reasoning for it and many more details here.

Level AAA

There are five new success criteria for level AAA, which is the highest conformance you can achieve. The new success criteria are:

Level AAA is the least common level of conformance but I still think there are success criteria here that should be implemented. Things like re-auth, timeouts, and target size aren’t that hard to implement and probably are already implemented in your app since these are solid UX patterns.

1.3.6 Identify Purpose

This success criterion looks hard to implement properly. The intent is for your site or app to support personalization and preferences. Here are the examples they give in the understanding document:

A website uses ARIA landmarks to identify the regions of the page, and users can hide areas that are not the 'main'.
The links in the navigation of a website are marked-up so that users can add their own icons.
Icons on a website use are marked-up so that the user can substitute their own icon set into the page.

You can see how this one might be pretty hard to do right.

You can read the reasoning for it and many more details here.

2.2.5 Re-authenticating

When an authenticated session expires, the user can continue the activity without loss of data after re-authenticating.

You can read the reasoning for it and many more details here.

2.2.6 Timeouts

If there is a timeout present on the page, there should be a warning shown before the session ends. You should provide a warning for anything that might cause data loss due to the session ending.

These are usually implemented with a modal that pops up warning you that time is running out. Make sure that modal is accessibile too!

You can read the reasoning for it and many more details here.

2.3.3 Animation from Interactions

If you have animations on your site the user should able to be disable them. Unless they are essential. What WCAG defines as an “essential” animation:

A web application provides a feature to author animated sequences. As part of this tool, the author needs to preview the animation.

You can read the reasoning for it and many more details here.

2.5.5 Target Size

The target size of pointer inputs (mouse, pen, touch contact) should be a minimum 44px by 44px unless:

Equivalent: there is another target on the same page that provides the same action that meets this success criterion.
Inline: the target is in a sentence or block of text
User Agent Control: the size of the target is controlled by the user agent & is not modified.
Essential: the specific presentation is required for the target to function.

You can read the reasoning for it and many more details here.

2.5.6 Concurrent Input Mechanisms

I really like this addition to WCAG. This success criterion says you should make sure your content can be interacted with through different modes of input.

The easiest example to give would be making sure your site or app works if someone pairs a wireless keyboard to their phone to navigate. Another example would be making sure your site can be interacted with from a laptop with a touchscreen. This exact example came up on Twitter over the weekend too!

I like this one because my mom is 100% blind and uses her iPhone with a keyboard a lot of the time. It’s just easier than swiping sometimes.

You can read the reasoning for it and many more details here.

Wrap up

Whew! That was a lot to unpack. One of the biggest struggles I had when getting started in accessibility was finding resources that digested the WCAG spec into more concrete examples. Jumping right into specifications as a first step can be daunting. With that said I hope this post was enough for you to get familiar with the changes to WCAG and what they mean in simple terms.

I think it is worthwhile for you to spend some time reading through a couple of the documents that explain the reasoning for these success criteria. One of the realizations I’ve had when reading through the spec is WCAG is a set of guidelines for improving your site’s UX. Accessibility really is all about making sure your site or app can be used by anyone, which serves as an excellent foundation to build upon for your site’s UX.

If you or your team need any help getting up to speed with WCAG 2.1 reach out to us! If you would like to ask any questions or continue this conversation feel free to reach out to me on Twitter.

Functional Rephrasing Using an Async Function Functor

Charles Lowell — Tue, 03 Jul 2018 16:00:00 GMT

One of my favorite things about the the math framework is that it helps you look for (and also find) patterns of composability around you in your programming world. At the time I wrote:

As you start to perceive the common nature in things like Promise, Array, and Observable, you begin to re-perceive all of the objects in your world through this new prism. You ask yourself “Is this a Functor?” and you’re shocked by how often the answer is yes. Plain JS Object? Yes. DOM Element? Yes. Express middleware? Yes. React Component? Also yes.

It turns out that Functors are everywhere, and armed with funcadelic.js at your side, you stand ready to harness their power. In fact, just yesterday I found just such a one: The JavaScript AsyncFunction. And today, I'm going to show you how we can use the fine fact that async functions are Functors in order to refactor some code to be more composable. In other words, we'll functionally rephrase it as a Functor.

Let's say we have an async function that fetches a url for us:

async function get(url) {
  return await fetch(url);
}

And now we want to assemble a data transformation pipeline around it. For argument's sake, let's say our process will look like:

-> get the data
-> parse as JSON
-> normalize the keys in the resulting json
-> look up the canonical model in a local data store.

As a first pass, we could write a transform function that takes the url and applies each of these steps.

async function transform(url) {
  let response = await get(url);
  let data = response.text();
  let json = JSON.parse(data);
  let normalizedKeys = normalizeKeys(json);
  let model = lookup(normalizedKeys);
  return model.
}

And if we wanted to stop there, life would go on. However, we could also declare that AsyncFunction is a Functor and leverage the flexibility that it provides. Here is the definition of the Functor instance for async functions.

import { Functor } from 'funcadelic'

// AsyncFunction is not currently a global object. :/
const AsyncFunction = Object.getPrototypeOf(async function(){}).constructor

Functor.instance(AsyncFunction, {
  map(fn, asyncFn) {
    return async function(...args) {
      return fn(await asyncFn(...args));
    }
  }
})

What this says is: whenever you map a function fn over an async function, what you get is a new async function that just invokes the old async function, awaits its return value, and then applies fn to it before returning.

Using the funcadelic chaining syntax this let's us define the transform method, not as a monolithic function, but as the combination of a bunch of smaller functions.

import { chain as $ } from 'funcadelic';

const transform = $(get)
  .map(response => response.text())
  .map(data => JSON.parse(data))
  .map(json => nomalizeKeys(json))
  .map(normalized => lookup(normalized))
  .valueOf();
}

We start with the first async function get, and then map a plain function over it that takes the response and returns its text. The result of that is an async function over which we can then map a plain function that just parses the JSON. The result of that is another async function over which we can map key normalization, and so on....

Just as when we declared it explicitly, transform is an async function because the result of mapping an async function always returns another async function. That's just how Functors work.

Notice also that for our mapping functions that just invoke single argument functions, we can even leave out the arguments:

const transform = $(get)
  .map(response => response.text())
  .map(JSON.parse)
  .map(nomalizeKeys)
  .map(lookup)
  .valueOf();
}

Which ever style you choose, the neat thing is that we're building up an async function out of a bunch of regular plain-old JavaScript functions with nothing async about them. This gives us a lot of flexibility if, for example, we want to add a step to the pipline.

Let's say that we want to add a de-duping function in between the key normalization and the lookup from the data store. All we have to do is insert it into our pipeline.

const transform = $(get)
  .map(response => response.text())
  .map(JSON.parse)
  .map(nomalizeKeys)
  .map(dedupe)
  .map(lookup)
  .valueOf();
}

Pretty neat! But there's one more thing I have to show you, and I think you're going to like it..

We can use the Functor law which basically says that you can either map each function individually, or you can compose them all together and map that function once. In other words:

import { map } from 'funcadelic';
import { pipe } from 'ramda';

let pipline = pipe(
  response => response.text(),
  JSON.parse,
  normalizeKeys,
  dedupe,
  lookup
);

const transform = map(pipeline, get);

This is all not to say that there's a single way to do it. Far from it! But with the AsyncFunction Functor in your pocket, you've got options on your side!

Photo by Jason Blackeye on Unsplash

Big Testing in React

Wil Wilsman — Wed, 13 Jun 2018 00:00:00 GMT

Does my application work in real life?

This is a big question, and big questions need big answers. How do we answer that question, though? With big tests. When we write and develop our applications, it's with the intent that somebody will use it. We write tests so we can be assured that everything in our app works as expected and is bug free. But how confident are you in your tests?

A real person using your app is going to use a real browser; do your tests test your app in a real browser? And not everybody uses the same browser; do your tests test your app across multiple, different browsers and devices? What about the network? Chances are your app talks to the network, and your tests should account for this too, right? Not to mention a person isn't going to interact with your app on a component level. They're consuming the entire app , using all of it together. All of your components working with each other to create an experience.

If your tests don't test the app in the same way a person would use it, how confident can you really be in them?

Where do we even begin to test our apps like this? There are tools like Jest, but those tests don't run in a real browser. There's also Cypress, but as of this writing, you currently can't use it outside of Chrome. There's also a looming thought that "user tests are slow." Do they have to be?

Anatomy of a Big Test

When we talk about testing the entire app in a way that a person would interact with it, we're of course talking about acceptance tests, or user tests, or end-to-end tests. These types of tests have many names, but it all refers to the same process of testing your app as if you were an actual user.

The life of an acceptance test is pretty simple. First, our app is mounted and we wait for it to load. Then, some interaction is performed, such as filling out a form, or clicking a button; something a person would actually do in our app. Finally, we make some expectations about the state of our app after the interaction is performed. Maybe this is a success message, or some other confirmation. If a user does something in our app, they probably received some feedback. We want to test this interaction and ensure that it works as they would expect it too.

When a person is interacting with your app, they're using their mouse and keyboard which is translating to browser events that your app responds to. They're not calling methods and expecting them to return certain values, they're actually clicking things and expecting to achieve results. This is what our app's tests should do too: send browser events and assert that there was feedback.

But how do we actually do all of this?

The Big Setup

Luckily, it's not nearly as hard as it sounds. How we interact with our app should be so identical to how our tests interact with our app that the only difference is the entry point.

In development, our entry point imports our app, mounts it, and that's it! The app is ready for us to start interacting with. If we're working on a certain feature, we navigate directly to the part of our app containing it.

In testing, our entry point imports all of our tests and runs them. It's each test that mounts and interacts with our app. But from our app's point of view, nothing is any different than if it were being interacted with by a person. Only, in our tests, we can reset our app when necessary to make sure we're working with a clean state.

Let's get to some actual code! For example's sake, let's say our app is built using React and Webpack with babel-loader, style-loader, and html-webpack-plugin. We'll also be using Mocha and Chai to write our tests. The app in our example is a simple countdown app that accepts a date input through some custom spinner inputs and displays the time until or elapsed since the specified date.

Here's a quick look at our example app in action (direct link):

Our application's entry point looks like this:

// src/index.js
import React from 'react';
import { render } from 'react-dom';
import App from './app';

render(<App />, document.getElementById('root'));

If you have other necessary app code in your entry point, consider moving it into a reusable module. You could even have a Root component with the logic written in lifecycle hooks. The point is: during development, or in production, our app is only mounted once; for our tests, we need to be able to remount our app before each scenario.

Our testing entry point will look something like this:

// tests/index.js

// mocha doesn't support es modules, so we import the pre-compiled assets
import 'mocha/mocha.js';
import 'mocha/mocha.css';

// other setup could be done here too, such as registering chai helpers
mocha.setup('bdd');

// we import our tests using webpack's require.context
// if we just use `import`, it will be hoisted above mocha's bdd setup
const requireTest = require.context('.', true, /-test/);
requireTest.keys().forEach(requireTest);

// run mocha
mocha.run();

It's each test's job to mount and interact with the app, so all we need to do here is import all of our tests and run them. We'll go over what our tests will actually look like in just a minute. For now, let's pretend that our tests just do their aforementioned jobs. You might be able to figure out where I'm going since I've been mentioning entry points. We're going to use our existing Webpack config to bundle our tests.

// webpack.config.js
module.exports = (env) => ({
  entry: env.testing ? './tests/index.js' : './src/index.js',
  // ... other webpack config
});

We're using Webpack v4, which allows us to [export our config as a function]([Configuration https://webpack.js.org/configuration/configuration-types/#exporting-a-function). The first argument of this function, env, is populated by the CLI argument of the same name. This way, when we start our app, we can just call yarn start --env.testing to switch to our testing index. This new bundle will run all of our tests for us, and we can open this bundle in any browser on any device to run our tests there as well!

Note: in our specific case, Mocha's HTML reported requires a #mocha container in the DOM to output it's report. We're using html-webpack-plugin along with html-webpack-template, so we can just change the appMountId of the template when we're in a testing environment.

Okay, so we know how to bundle our tests, but how do we actually write them?

Writing Big Tests

If we reflect back on our anatomy lesson, our tests do three important things:

Mount/Remount our app
Interact with our app in a meaningful way
Assert that something happened

To successfully do all of these things, we'll be using a set of libraries developed specifically for testing big. The BigTest suite of libraries and framework extensions help to answer the big question: does my application work in real life? We'll touch on a few of those libraries in the rest of this article, but there are some we won't get to, and more planned for development.

Mounting Our App

First, to mount our app we can just use render from react-dom. But according to the docs: "If the React element was previously rendered into the container, this will perform an update on it and only mutate the DOM as necessary to reflect the latest React element." However, this is not exactly what we want. While this does mount our app, we actually want to mount a fresh instance of our app every time. To do this we can combine render and unmountComponentAtNode, which will completely remove a component from the DOM after calling the appropriate lifecycle hooks.

This is pretty simple to do ourselves in our own mount helper, but to spare us from even more DIY architecture, we can just utilize the helpers from @bigtest/react which already do this. It also provides some other useful things, such as setting up an in-memory history object for our app to provide to React Router, and binding this history object to other helpers that allow us to navigate our app from within our tests.

This is how we would use @bigtest/react to mount our app and visit a route:

import { setupAppForTesting, visit } from '@bigtest/react';
import { expect } from 'chai';

import App from '../src/app';

describe('mounting our app and visiting a route', () => {
  beforeEach(async () => {
    await setupAppForTesting(App);
    visit('/2019/01/01');
  });

  it('shows the date', () => {
    let $date = document.querySelector('[data-test-countdown-target]');
    expect($date.textContent).to.equal('January 1st, 2019');
  });
});

We can reduce some boilerplate for all of our tests by creating a helper file and combining the React helpers with Mocha hooks. This way we also won't have to import our app into every test file.

// tests/helpers.js
import { setupAppForTesting } from '@bigtest/react';
import App from '../src/app';

export { visit, location } from '@bigtest/react';

export function setupApplicationForTesting() {
  beforeEach(async function() {
    this.app = await setupAppForTesting(App);
  });
}

We could have other helpers in this file, or do further setup such as creating a mock server. Once we have an easy way for our tests to mount our app on demand, we can interact with it.

Interacting With Our App

Again, we're not testing that some function returns some value or calls some other function. We want to interact with our app like a real person would.

To do this, we're going to use @bigtest/interactor to send real browser events to our app. I won't go into too much detail here, but interactors are a very powerful, composable, way to interact with the DOM. Interactors will wait for elements to exist before interacting with them, so we don't have to worry about timing our tests correctly to sync up with any run loops.

If you're familiar with the page object pattern, you'll know that they can abstract the internal structure of a page away from the user-facing functionality. They also reduce a lot of boilerplate around the UI in our tests, so that if changes are necessary, only the page objects need to be updated. You can think of interactors as composable page objects for modern components.

Let's create an interactor for the custom spinner input in our app:

// tests/interactors/spinner.js
import { interactor, clickable, value } from '@bigtest/interactor';

export default @interactor class SpinnerInteractor {
  // common helpers provide easy to define interactions
  prev = clickable('[data-test-spinner-prev]');
  next = clickable('[data-test-spinner-next]');
  value = value('input');

  // complex interactions can be expressed as methods
  select(option) {
    let self = this;

    // for each letter in `option`, send a keypress event
    // that will trigger the spinner's typeahead feature
    for (let key of option) {
      self = self.trigger('keypress', {
        charCode: key.charCodeAt()
      });
    }

    // methods that return a new instance of itself are chainable
    return self;
  }
}

And we can use this interactor like so:

// interactors are lazy, so we can initialize them without anything
// actually existing in the DOM yet
const year = new SpinnerInteractor('[data-test-year-field]');

// when accessing properties, an error might be thrown if the element does
// not exist; this can be more useful than typical expectation failures
expect(year.value).to.equal('2019');
//=> Error: unable to find "[data-test-year-field]"

// methods return new interactors that can then be chained from one another
let selectYear = year.select('2019')
let selectYearThenPrev = selectYear.prev();

// interactions do not start until `interactor.run()` is called, or when
// used with the async/await syntax; they will resolve once the element
// exists and has been interacted with, or when the timeout ellapses
await selectYearThenPrev;
await year.next();

We can then take these smaller interactors and compose them to create more complex interactions.

// tests/interactors/form.js
import { interactor, scoped } from '@bigtest/interactor';
import SpinnerInteractor from './spinner';

export default @interactor class DateFormInteractor {
  // the default scope allows us to omit the selector when calling `new`
  static defaultScope = '[data-test-date-form]';

  // interactors are composable and nested methods return an instance
  // of the top-most interactor for chaining
  year = scoped('[data-test-year-field]', SpinnerInteractor);
  month = scoped('[data-test-month-field]', SpinnerInteractor);
  day = scoped('[data-test-day-field]', SpinnerInteractor);
  hour = scoped('[data-test-hour-field]', SpinnerInteractor);
  minute = scoped('[data-test-minute-field]', SpinnerInteractor);
  submit = scoped('button[type="submit"]');

  // getters can also be used for computed properties
  get value() {
    let date = `${this.year.value}/${this.month.value}/${this.day.value}`;
    let time = `${this.hour.value}:${this.minute.value}`;
    return `${date}@${time}`;
  }
}

You'll notice our use of data-test-* attributes. This is because, if using element IDs or CSS selectors, updates to the app's design or markup mean your tests will most likely break. By using data attributes, we can make our tests and interactors resilient to future changes. We can also use a babel transform like babel-plugin-remove-jsx-attributes to remove our testing attributes in production.

Now that we've successfully set up some interactors, we can use them to start interacting with our app in our tests.

Making Assertions About Our App

Making assertions sounds pretty straightforward, but since real apps typically have some async operations associated with them, making assertions at the correct time can be cumbersome and headache-inducing.

For example, let's say we need to test that when a user clicks a button they are shown a random message. When does the message appear in the DOM so that we can make an assertion about it? After the user clicks, sure, but how long after the user clicks? In React, it's whenever the render loop gets around to rendering that specific message. What if the app makes a network request for this message first? Well, now we need to also wait for a response before making our assertion.

Interactors wait for elements to exist in the DOM before interacting with them, but can we do the same thing with our assertions? Well, yes we can! Interactors use @bigtest/convergence under the hood, and we can utilize convergences for assertions too.

What's a convergence? To put it simply, it's a pure assertion that can run repeatedly until either it passes, or a timeout has been exceeded. And pure assertions are expectations that do not cause side effects.

import { when } from '@bigtest/convergence';

// will only continue once the expectation is true
await when(() => !app.state.isLoading);

// works when throwing errors as well
await when(() => {
  expect(foo).to.equal('bar');
  expect(baz).to.be.undefined;
});

Convergent assertions will be called several, maybe even hundreds of times, depending on the timeout and if the assertion ever passes. So having side effects in our assertions could cause unintended behavior that may also result in some serious slowdown.

In regard to our tests, if we make sure all of our assertions are all pure by keeping our interactions inside of hooks, we can use @bigtest/mocha to automatically turn all of our it statements into convergences for us.

import { describe, beforeEach, it } from '@bigtest/mocha';
import { expect } from 'chai';

import { setupApplicationForTesting, location } from './helpers';
import DateFormInteractor from './interactors/form';

describe('Date Picker', () => {
  const form = new DateFormInteractor();

  // for our example app, this function also stubs the date to a constant value
  setupApplicationForTesting();

  it('defaults to the current date', () => {
    expect(form.value).to.equal('2018/1/1@0:0');
  });

  describe('submitting a date and time', () => {
    beforeEach(async () => {
      await form
        .year.select('2019')
        .hour.select('12')
        .minute.select('30')
        .submit.click();
    });

    // if this fails, @bigtest/mocha's `it` will keep rerunning until it
    // passes or a timeout has been exceeded
    it('goes to the date countdown', () => {
      expect(location().pathname).to.equal('/2019/01/01/12:30');
    });
  });
});

And there we go! We've officially written some big tests that properly interact with our app how a user would.

The Big Finale

Alright, so we've written our tests and we can run them just by running our app in a testing environment. Our tests interact with our app just how a user would, and all of our tests pass with flying colors. But we have real projects with CI/CD pipelines and these tests need to be automatically run, reported, and tracked for us.

Thankfully again, we don't have to do this ourselves. Karma is a framework agnostic test runner that can automatically launch browsers and generate reports for us. It has plenty of plugins to work with the browsers and frameworks we're already using. For our test suite, we're going to be using karma-webpack, karma-mocha, and karma-mocha-reporter. To launch our browser, we'll use karma-chrome-launcher.

There are a couple things we need to change first to get our Karma plugins working smoothly:

karma-webpack takes a Webpack config, but will disregard our entry and use the file defined in our Karma config instead. This means some of the testing setup we did in our Webpack config won't be necessary anymore.
karma-mocha will automatically set up Mocha and run our tests for us, so we'll need to remove the Mocha imports and initialization we manually did in our testing index.

Our new testing index will now look something like this:

// tests/index.js
// other setup could be done here too, such as registering chai helpers
const requireTest = require.context('.', true, /-test/);
requireTest.keys().forEach(requireTest);

And our Karma config will look like this:

module.exports = (config) => {
  config.set({
    frameworks: ['mocha'], // sets up the Mocha framework automatically
    reporters: ['mocha'], // reports back to the CLI formatted like Mocha
    browsers: ['Chrome'], // automatically launches chrome to run our tests

    files: [
      // karma-webpack will watch our files
      { pattern: 'tests/index.js', watched: false }
    ],

    preprocessors: {
      // tells Karma that we'll be using Webpack to process this file
      'tests/index.js': ['webpack']
    },

    // Mocha reporter options
    mochaReporter: {
      showDiff: true
    },

    // our webpack config exports a function
    webpack: require('./webpack.config')(),

    // webpack dev middleware options
    webpackMiddleware: {
      stats: 'minimal'
    },

    // for more Karma config options, check out the documentation
    // http://karma-runner.github.io/2.0/config/configuration-file.html
  });
};

With all of that set up, we can run karma start to launch our browser and automatically start our tests! For our CI/CD integration, we can add the flag --single-run to exit after running our tests in the specified browsers. To add other browsers, we simple just have to install other launchers like karma-firefox-launcher or karma-safari-launcher. We could even use any of the browsers and devices available on BrowserStack via karma-browserstack-launcher!

With a little bit of manual setup, and through testing big, we've answered the question: Does my application work in real life? However, we've only scratched the surface in this article. We glossed over some things like convergences and interactors, and we didn't even get to touch on server mocking with @bigtest/mirage.

Our countdown app lives in the BigTest examples repository where you can check out the rest of our tests and the entirety of our configuration files. There will be more examples to come, including other frameworks and more complicated test setups. Keep an eye out in the future for more BigTest blog posts and guides, and feel free to follow and contribute to all of the projects over at the BigTest GitHub organization.

Connecting a Slack Channel to Gitter

Robert DeLuca — Mon, 30 Apr 2018 14:00:00 GMT

Slack communities for open source projects have become very popular over the past of couple years. Slack has taken hold because it makes collaborative chat really easy and accessible for the masses, something IRC wasn’t able to achieve.

Slack is amazing for teams working together but there are downsides to using it for open source communities. One of those downsides is they’re closed off to search engines. When someone gets help with an issue it’s forgotten about. Another issue is that they are usually free Slack communities, which means they have limited archives. Once the limit is reached, messages are deleted.

Recently we have started looking into ways to make sure we’re available to the community for our open source projects. One of the ideas was to have Slack channels where we could collaborate with the community, but considering the limitations stated above, we couldn’t pick Slack.

In our search for something with Slack-like qualities, we found Gitter, which checks pretty much all of the boxes! One problem though: we knew that if we had two different chat apps we would end up forgetting about one (and it was going to be Gitter). If only there were a way to combine them…

What if they could both be the Sameroom?

There’s a service out there that allows you to connect one chat app to another called Sameroom (see what I did there? (☞ﾟヮﾟ)☞). We decided to give it a shot and, I have to say, it’s pretty damn neat. We connected our internal open source Slack channels to the Gitter chatrooms we set up. Now we will get messages posted in our Slack channel whenever someone posts in our Gitter chatroom.

We decided to only have the chat go one way. The internal Slack chat won’t post to Gitter because we’re not crazy about the way it integrates. Every post in our Slack channel comes from Sameroom bot in the Gitter chat. If it was easier to distinguish who those posts were coming from we’d absolutely enable it. Gitter covers this limitation in this blog post.

Other than that one issue, this integration seems to work pretty well! We’ve set it up for two of our projects and are looking to add more in the future. Come chat with us in the Frontmacs or BigTest Gitter!

Math is Just Another Framework

Charles Lowell — Mon, 19 Feb 2018 13:00:00 GMT

Monads. Amirite?

And Ugh... Monoids? 🙄 please 🙄

What a bunch of complicated gobbledygook. I mean first off, you have to have like, a PhD in mathematics just in order to get started with crap like that, and then if you do decide to put in the work, what's the payoff? Look, I just make webapps here. The only return I'd ever see on that type of investment is maaaayybe the dubious honor of hanging out with developer cliques that fetishize abstruse words like "Endofunctor" and "Isomorphism." They're so exclusive anyway, and that's not what I'm about. I build real things, and so this isn't for me.

Does any of that sound like an internal dialogue you may have had with yourself at some point in the last few years? I know it certainly captures the color of my thinking on the subject of formal functional programming during that time. Sure, I took half-hearted swipes at understanding things like Monads here and there, but it never really stuck. And more often than not, I was inclined to chalk it all up to sour grapes... That is, of course, until I realized something amazing. I realized that contrary to what I was brought up to believe, formal FP wasn't something that would forever lie just beyond my reach.

Imagine my surprise in realizing that not only was this not true, but in realizing that I had actually been an FP super hero all along; descended from a long line of FP super heroes. And do you know what the most amazing thing is? Whether you know it or not, you are too. Don't believe me? Just listen.

It all began with a Functor

A Functor? Are you kidding me?

I know, I know. Even now just saying it still makes me want to giggle. But if you're still doubting your own powers, then Functor is where the journey begins. That's where it began for me. Because you know what? If you code for a living, then odds are you use Functors every coding minute of every coding day. You just haven't realized it yet.

So Functor is the name, and the concept it references is "mapping over a structure". There's nothing more to it than that. Functors are things that can be mapped. End of story. Arrays can be mapped: Arrays are Functors. Observables can be mapped: Obvservables are Functors. So guess what? Like it or not, if you are comfortable mapping Arrays, then you are also comfortable working with Functors.

Mapping an Array always returns an Array of the same length, but with different values at every index. Mapping an Observable always returns an Observable with the exact same sequence and timing of states, but with different values inhabiting each state. The general pattern is that mapping doesn't change the object's structure.

At its core, an Array is nothing but a sequence of indexed placeholders that contain values. That is its structure, and that's exactly what doesn't change when you map it. The structure of observables is the timing and sequence of states coming through a stream, so it's the same timing sequence that remains intact when you map it. So no matter the Functor, when you map it, it returns a new Functor with the exact same structure. In other words: S -> S.

But you knew already that mapping an Array of the same length always returns an Array of the same length because you've been working with them for as long as you've been writing code.

Crouching Promise, Hidden Functor.

So yeah. I know how to map Arrays. Big deal. How does assigning some new name to what I'm already doing actually help me in any tangible way? It seems an awful lot like some boring thing to remember.

Well, this is where the plot takes an unexpected turn. This is the part in the story where a chance encounter with a secret Functor plants the suspicion that perhaps you've been lied to your entire life by the ones that raised you. Sure, they meant well, and all they ever really wanted to do was to save you sweat and tears, but in the end the only thing they actually accomplished was to delay your inevitable confrontation with destiny.

Here's your first clue: Have you used and/or thought about Promises lately?

Promises don't have a map method, and yet they are nevertheless Functors. They're the functoriest of Functors that ever were, and their functorality is as constant and sure as the sun rising in the east and setting in the west. But how can this be if there isn't any way to map them? The answer is of course that you can map them. You just have to use the then method to do it. It's this alternatively named mapping function that makes Promise a secret Functor.

It makes sense when you think about how then works. You start with a Promise of some value that will resolve at some later time, and you end up with another Promise that has the exact same structure as the original in that it will also resolve at the exact same time as the first. The only difference is that the value has been transformed by the passed in function. Once you come to recognize it, you see that it's a clear stamp out of the Functor pattern, and that the method might as well have been called map instead of then.

And that's when you begin to wonder "what if it had been named map instead of then, and explicitly identified as a Functor from the get-go?" Learning to work with Promises was tricky enough, but what if you'd been able to leverage the fact that the greater part of working with them was strongly analogous to working with Arrays? What if you'd been able to apply everything you knew about mapping Arrays and bring it to bear on Promises? How much time and mental exhaustion would it have saved? Looking back at my own experience, it would have been extraordinarily helpful to have sidestepped the process of trial and error that happened as my brain settled around the semantics of Promise.

Here's the rub. The method name then is a reference to the timing of Promise resolution, which of course seems reasonable at first. After all, it's a good thing for an API to be concrete and descriptive right? The hidden tradeoff is that the name then orients the API around what makes Promise unique, instead of what makes Promise literally like every other Functor. And there are thousands of them.

They inhabit so many of the programming structures that you use every single day, that once you're aware of them, they will begin to reveal themselves to you. As you start to perceive the common nature in things like Promise, Array, and Observable, you begin to re-perceive all of the objects in your world through this new prism. You ask yourself "Is this a Functor?" and you're shocked by how often the answer is yes. Plain JS Object? Yes. DOM Element? Yes. Express middleware? Yes. React Component? Also yes.

And then it hits you.

Math is Just Another Framework

If you're using Ruby on Rails, and someone tells you about a Controller, it doesn't matter if it's a controller for products, or users, or blog posts. The mere fact that it's a Controller gives you most of what you need to know about its role and how you can set to work with it to build the larger application. By the same token, if someone tells you that some object is a Functor, it doesn't matter if it's a list, or a tree, or a stream. The mere fact that it's a Functor gives you most of what you need to know about its role and how you can set to work with it to compose it into larger structures.

I use Ruby on Rails as an example to illustrate this point because the framework is synonymous with the concept of convention over configuration, and that's the fundamental dynamic at play here. But whereas Ruby on Rails is a framework for web applications, formal FP grounded in Category Theory is a framework for composable APIs.

Like learning any new framework it's an investment. There are patterns to internalize and there is abitrary jargon to become acquainted with, but don't be intimidated. Every framework has its concepts that seem opaque at first, but you can learn them all one piece at a time. And once you're familiar with them, the big return is that you can quickly use typeclasses like Functor to build APIs that compose well with each other out-of-the-box at zero extra cost. And the best part is that it's a framework that's 100% portable across runtimes. Whether you're using JavaScript, Ruby or Python it "just works" every time.

If you do happen, however, to be working in JavaScript, you're in luck. I wrote a library just for you called funcadelic.js that brings the magic of typeclasses into the reality of everyday JavaScript development.

Because Functor is just the beginning. Like a programmer learning Ruby on Rails can start with Controller first, and then move on to Router, Model, and Middleware, so you can you move on from Functor (at your very own pace) to using Applicatives, Monads, and Monoids.

So go for it! I remember the first time I took a tree class I'd been working with and started thinking of it as a Functor. All of a sudden, the brain-hurty, recursive aspects of the tree were abstracted into the map method and I was able to relate entire trees to each other with nothing more than a simple function. I cannot describe the feeling other than it felt like flying. The ground below was all very familiar because I'd walked it back and forth many times over the years. The only difference was that now I had the perspective of ten thousand feet and the airspeed of a jetliner.

You know the ground too. You walk it every day, and if you're willing to learn the math framework, the same feeling is waiting for you.

I’m Charles Lowell, and I build UI for a living at Frontside. If you enjoyed this post, I’d love to hear from you. You can give me a shout on Twitter where I'm @cowboyd, or drop me a line at cowboyd@frontside.com

Quickly Build Component Libraries with Storybook

Jeffrey Cherewaty — Fri, 29 Dec 2017 23:00:00 GMT

As long as humans have been building things, we've tried to break up the work. When we can carve out smaller tasks, we can share the work among more people and focus on the quality of each little piece.

Tools and technologies have reached a point where nearly everyone building for the web is thinking in components. But when you've got a large team all creating and using components, how do you keep the usage consistent? Designers have had a tool for that problem for decades. Style guides set down rules and guidelines for projects, so they spoke with a consistent look and voice.

By combining a style guide with a library of components, you can create a comprehensive design system.

Enter Storybook

Prominent companies that have built large design systems usually have to build all new tools for their component libraries. When you have multiple templating languages to deal with and dozens of different product teams, your design system has constraints that no out-of-the-box solution provides.

At Frontside, we're frequently working on brand-new projects instead, so we push open-source solutions that let developers immediately focus on delivering features. For component libraries featuring Vue or React, there's now a one-size-fits-most tool: Storybook.

Writing Stories

With Storybook, developers create "stories" that illustrate and describe different versions of each component. In practice, it feels a lot like writing tests: what are all the possible iterations of this component? Because it aligns with test-driven development philosophy, Storybook makes a great base for adding visual regression testing.

import React from 'react';
import { storiesOf } from '@storybook/react';
import Checkbox from './Checkbox';

storiesOf('Checkbox', module)
  .add('Basic Usage', () => (
    <Checkbox
      onChange={() => {}}
      checked=true
      label='Check Me'
    />
  ));

Storybook Docs: Writing Stories

Addons

Storybook's addon architecture is where the benefits of having an open-source project really pay off. There's a whole ecosystem of pre-built addons, along with the ability to create your own.

README addon

The codebase we're working with already had great documentation for many of its components. With storybook-readme, just a couple lines of configuration had each component's README appear in the Storybook environment

import Readme from './README.md';

storiesOf('Checkbox', module)
  .addDecorator(withReadme(Readme))

Add a knob

Another useful Storybook addon: storybook-knobs. Component props can be edited within the Storybook UI.

import { withKnobs, text } from '@storybook/addon-knobs';

storiesOf('Checkbox', module)
  .addDecorator(withKnobs)
  .add('Basic Usage', () => (
    <Checkbox
      onChange={() => {}}
      checked=true
      label={text('Label', 'Check Me')}
    />
  );

Add an action

storybook-actions help drive component action development. Component actions get stubs that show up in the Storybook UI.

import { action } from '@storybook/addon-actions';

storiesOf('Checkbox', module)
  .add('Basic Usage', () => (
    <Checkbox
      onChange={ action('checkbox-toggle') }
      checked=true
      label='Check Me'
    />
  );

Evolving Tools

Storybook is a big landmark for developing component libraries for the web, and we're looking forward to its evolution, and eventually the next set of tools for design systems.

Check out this Lunch and Learn to see Storybook in action.

Need help creating your own design system? Reach out to Frontside.

How Healthy Is Your Build?

Joe LaSala, Ginger Whalen, Elrick Ryan — Fri, 04 Aug 2017 12:00:00 GMT

When’s the last time you went in for a checkup with your family doctor? Do you know what your cholesterol level is? Your blood pressure? It’s unfortunate that many people can’t answer those questions quickly and honestly. It usually takes a health scare to prompt people towards that dreaded waiting room with the stale air and old magazines.

Regular checkups are an essential part of healthy living, so why do we put them off? Why do we ignore symptoms? Is it because we’re too busy? Are we afraid of what we might find out?

We all know what it takes to stay healthy: be proactive, practice good habits, monitor your health closely. We also know that these things are much easier said than done. I mean, come on! Who has time to take care of every little wrinkle in the moment, right? Thankfully, applying these healthy practices to your application’s build health is far less arduous. That you can handle, and we can show you why it’s worth it. Let’s get out of the red and into the green!

Our goal? Master builds that are validated by a comprehensive test suite and pass 100% of the time. In other words, a perfect world. But hey, it’s easier than eating more kale, cutting out the bacon, and exercising three times a week.

So, let’s do a quick checkup. Don’t worry, no stethoscopes: this is a build check up. Off the top of your head, do you know how many times your application failed to build last week? If that number is greater than zero then you just failed your checkup. If you don’t know the number, you may need to see a specialist.

Fear not! We can write you a prescription that will get that number out of the danger zone. Side effects may include happier engineers, happier users, and an increase in code and product quality.

In order to make a diagnosis, we first have to ask you some questions...

About your test suite...

What is your current test coverage? Do you enforce a coverage threshold?
Are you writing good tests? There are many articles devoted to the finer points of what makes a test “good”, so we’ll leave it up to you to find a definition that makes sense for your team. Whatever that definition ends up being, make sure you question it regularly and revise it if necessary.
Are you writing appropriate tests? Do you understand the difference between unit tests, integration tests and acceptance tests? What about “end-to-end” tests or “smoke” tests? The nomenclature around these concepts is often overloaded and the concepts themselves are frequently conflated. Try not to get lost in the woods.
Are you testing your own code, or vendor code? Think about it. You may be surprised.

Hmm, OK. I see. Moving on...

How about your testing strategy?

Do you have one? What is it?
Is it just a concept in your procedures manual, or is it a reality every time new functionality is added to your application?
How often do you revise your strategy? How often do you set aside time to discuss it with your team? Is your testing process as important as development process?
How often does automated testing take a backseat to feature deadlines?

OK. Let me just mark that last bit in my chart...

Finally, with regard to continuous integration and deployment...

How often do you push new code to your production environment?
When you do push to production, how do you do it? Is there a clear process in place? How much of that process is handled manually? Every manual step in your deployment is susceptible to human error.
Does your deployment pipeline include a CI/CD service like Jenkins, CircleCI or TravisCI?
Is your CI/CD environment in relative parity with your development environments?

Alright, that should do it! Thanks for your honesty. Well, reader, if any of these points gave you pause, then I regret to inform you that your build may be in poor health! Luckily we caught it early. You can address this problem now by making a few lifestyle changes.

Good build health starts with a goode test suite. “System Integration Tests”, “User Acceptance Tests”, “Deployment Pipelines”... if you're not a software developer these terms might as well be medical jargon. For now we'll just give you the big picture: you have an application, people use the application. Your test suite needs to account for both sides of that relationship.

We recommend that you pay special attention to acceptance tests, which are tests intended to mimic real user interaction with the application. Why? While the benefits of simulating realistic user interaction should be obvious, somehow acceptance tests often get overlooked. For example, this survey showed that only 24% of respondents were testing their Ember addons with acceptance tests. That number should shock you. Out of all the varieties of tests that a developer can write, acceptance tests are those that most closely mimic user interaction. What’s more important than the real-world user interaction?

Finally, remember that a good test suite will benefit your employees, your users, and your stakeholders. In fact, an automated test suite is really just one expression of a concept that probably permeates your organization more than you realize.

When a product manager writes requirements for a new feature based on customer feedback, they’re writing "acceptance criteria". What are acceptance criteria if not a human-readable test suite?
When the engineer sits down to write the feature, the corresponding acceptance tests should closely resemble the acceptance criteria laid out by the product manager.
A QA specialist can use these specifications as a roadmap for testing the feature manually.
A technical writer can use this outline as a jumping-off point for user-facing help documents.

This same pattern appears, in one form or another, at nearly every level of the modern software development workflow. If you can understand this pattern as a “common language” between your departments, you will also understand how much the individual units of your team depend on each other. That means that a healthy build can lead to increased efficiency across the board, but neglecting your application’s build health can jeopardize your product at many levels.

Nobody likes surprises. Not at the doctor’s office, and not when deploying an application. So get on the right track to good build health today! Say goodbye to white-knuckle releases and on-call experts. Imagine the peace of mind that comes with comprehensive test coverage. Imagine all the time you'll save with a realiable, automated deployment pipeline! That’s time that you could use to maintain your actual health. Take this advice and you’ll be able to hit that merge button with confidence, then hit the treadmill while your application hits the web.

Interested in talking more with us build health, or anything else software related? Get in touch on our website, or reach out to us on twitter at @thefrontside. We'd love to hear from you!

One Helpful Way to Think About JavaScript Decorators

Charles Lowell — Thu, 01 Jun 2017 12:00:00 GMT

Why Decorators?

It is an important question; one many tutorials rush past in their hurry to talk about the what and how. Of course, after being exposed to enough doses of what and how, you can usually piece together why, but in my experience it saves time to just hear it up front. I'd like to share one of the ways I've come to think about decorators and how it's helped me think about when to use them.

So like every good meta-discussion of JavaScript, let's start by talking about Clojure. Don't sweat it if you don't know Clojure, that's totally ok. The examples are very simple and only there to illustrate a bigger point.

Anyway, back to Clojure...

In Clojure, one way to associate a name with a value is to use the def form:

(def message "hello computer")

After that declaration, subsequent code can refer to the string "hello computer" just by using the symbol message instead.

The ability to associate names and values is a pretty handy thing in programmming, and it turns out you can do something very similar in JavaScript too. You might write something like this:

const message = "hello computer";

Another thing that Clojure has support for is function values: functions that can be passed as either an argument to, or the return value from, another function. You can declare a function value that adds its operands together with the following syntax.

(fn [x y] (+ x y))

Not to be outdone though, JavaScript also has support for first class functions, and it's syntax is even a little more concise than Clojure's too! This is it here.

(x, y) => { return x + y; }

Both of these are examples of anonymous functions. Function values that can be passed around as one-offs to other functions. It's handy, but sometimes it's not enough.

Often what we want is to associate a name with a function so that we can call it again and again. In the same way we associated the name message with the string value "hello computer" so that we could access it repeatedly just by referring to message, we can associate a function value with a symbol so that we can reach for it over and over just by remembering its name. To do that, we use the two constructs just introduced (def and fn):

;; associate the symbol `add` with the function to add
(def add (fn [x y] (+ x y)))

;; and now we can call it by name.
(add 1 1) ;;=> 2
(add 4 3) ;;=> 7

Just like before, a parallel construct exists in JavaScript. No suprises there.

// associate the symbol `add` with the function to add
const add = (x, y) => { return x + y; };

// and now we can call it by name.
add(1, 1) //=> 2
add(4, 3) //=> 7

But of course, this pattern of associating a name with a function is so prevalent that both Clojure and JavaScript have a special syntax to both declare a function and associate it with a name in one fell swoop. In Clojure you use defn (literally the conjuction of def + fn)

(defn add [x y] (+ x y))

And you're probably already familiar with how it's done in JavaScript with a function declaration.

function add(x, y) { return x + y; }

There's a point to this. I promise.

So I just explained named functions in both Clojure and JavaScript. Big whoop.

It's such a simple idea and an equally simple mechanism. And on the surface they seem equivalent in both languages, but there's one key difference between the way the named function syntax works in JavaScript and the way it works in Clojure. Both syntaxes declare a function value, and both associate that function value with a name so that it can be referred to later.

The difference is that in JavaScript the syntax is implemented by the JavaScript parser and interpreter, whereas in Clojure the syntax is implemented in Clojure.

A Macro Says What?

That bears repeating. In JavaScript, the syntax is implemented by the JavaScript parser and interpreter. More than likely that means baked in with some C or C++ code somewhere. Beyond the implementation, it means that there needs to be a specification for the syntax as part ECMA standard so that any of the multiple engines can be sure to get it right.

Figure 1: Put this in your C++ and smoke it

Contrast this to Clojure in which the named function syntax is nothing but more Clojure code. It's easy to understand not only for programmers, but also for any alternative implementation of Clojure (such as clojurescript). It's a nifty trick that saves time, adds clarity and eliminates ambiguity. How does it work? The answer is macros.

Macros are powerful, but they aren't fancy. A macro is just a function like any other function, and if you can work with functions, then you can work with macros.

There are two things that make a macro function special though. The first is that the type of its arguments and the type it returns are not general values but source code. That means it takes code in as its input, and returns code as it's output. Here's a couple of familiar functions along with their (pseudo) types for comparison. map takes an array as its input, and returns another array. sum takes an array as input, and returns a number. defn takes code input and returns more code.

map :: Array -> Array
sum :: Array -> Number
defn :: Code -> Code

The second thing that makes a macro function special is that it doesn't run when the rest of your program does; it runs when your program is being compiled. When the compiler encounters a macro, it calls it, and then replaces the macro with the code that got returned.

Figure 2: The arguments to a macro function are unevaluated source code

In fact, this is how the defn macro works. It literally replaces

(defn add [x y] (+ xy))

with

(def add (fn [x y] (+ x y)))

Figure 3: The defn macro takes code as arguments and returns more code

The point here is not to geek out too hard on Clojure macros, but to show just how much more you can do with so much less when your language has first class support for transforming code into other code. Large swaths of what we think of as Clojure syntax is actually defined with Clojure. The syntax it does have baked in works harder. There's no need to change the way the compiler works, and certainly no slog through a potentially contentious specification by committee.

Wouldn't it be nice if JavaScript let us do something similar? Wouldn't it be nice if we could extend JavaScript syntax with just a few lines of JavaScript code instead of years spent shit-posting on the internet?

Decorators to the rescue

The answer is that with decorators we can... almost. To be clear, decorators are not lisp macros. They're less powerful and they operate in a much smaller niche. But even so, they're an exciting development because there is just enough overlap with macros to make them helpful in the same kinds of situations.

That's because, like a macro, a decorator is also just a function. And also like a macro, it's a function that operates on code before the rest of your program sees it.

@decorator :: Code -> Code

Like a macro, we can use this to add the new syntax we always wanted to our JavaScript classes.

For example, I've often wished that there were an immutable keyword in JavaScript that made it impossible to change member data of an instance once it was constructed. Wouldn't it be great if you write a class like this?

immutable class Circle {
  constructor(radius) {
    this.radius = radius;
  }

  get area () { return Math.PI * this.radius * this.radius; }
}

let circle = new Circle(5);
circle.area //=> 25π

// can't do this!
circle.radius = 10 //=> Error!

I think that would be awesome, and if you feel the way I do, then you've got two options:

A. Begin your submission to the TC39 (you'd better get on it if you want to hit your deadline in three weeks!)

B. Do it now with decorators.

Decorators can help us get the syntax we want today because they let you transparently transform the code that you see into the code that you want. In the case of our immutable keyword, the code is a class. In other words, we want to take the class that we see (mutation allowed) and convert it into the class we want (no mutation allowed)

The class we see has this constructor:

class Circle {
  constructor(radius) {
    this.radius = radius;
  }
}

But the class we want has this constructor:

class Circle {
  constructor(radius) {
    this.radius = radius;
    Object.freeze(this);
  }
}

If we can write a function that takes the former and returns the latter, then we're in business. Luckily, this is pretty straightforward. In order to make an immutable version of any given class, we can extend it, and then make a call to Object.freeze after invoking the original constructor.

// @immutable :: Class -> Class
function immutable(Class) {
  class ImmutableVersion extends Class {
    constructor(...args) {
      super(...args);
      Object.freeze(this);
    }
  };
  return ImmutableVersion;
}

We now have a function that returns an immutable version of any class, and so now we can just use it as a decorator; almost like we were creating our own syntax.

@immutable class Circle {
  constructor(radius) {
    this.radius = radius;
  }

  get area () { return Math.PI * this.radius * this.radius; }
}

Our JavaScript interpreter will see this and transparently convert the Circle class that we see into a different one; one that prohibits any mutation of its member data after construction.

Decorators aren't just for entire classes. They can be used to enhance individual members of as class as well and transform them transparently into something more powerful. For example, since the area of the circle is a computed property, it won't be enumerable by default.

Object.keys(new Circle(5)) //=> ['radius']

That doesn't seem quite right. Shouldn't area be in there too?

What would be really nice if if JavaScript had a keyword to make any property enumerable. E.g.

{
  enumerable get area() { return Math.PI * this.radius * this.radius; }
}

It doesn't of course, but with decorators we can change that. We'll just write a member decorator: a decorator that operates on property descriptors instead of classes. We can write an enumerable function to take the property descriptor we see (non-enumerable) and return the property descriptor we want (enumerable). This is accomplished simply enough by returning a new descriptor exactly like the old one, except with the enumerable key of the descriptor set to true.

// @enumerable :: Property -> Property

function enumerable(target, key, descriptor) {
  return {...descriptor, enumerable: true};
}

With this in hand, we can mark the area property as enumerable, just as though the syntax had existed the whole time.

@immutable class Circle {
  constructor(radius) {
    this.radius = radius;
  }
  @enumerable get area() { return Math.PI * this.radius * this.radius; }
}

Declared this way, circles will let you know that they have an area in addition to a radius:

Object.keys(new Circle(5)) //=> ['radius', 'area']

Easy!

Decorators FTW

And that's it! My goal with this brief tour of decorators isn't to show you the ins and outs of how they work, but to share a helpful way to think about them and when they might be the right tool to reach for. While decorators are not lisp macros, it's useful to think about them as a spiritual grandchild in the sense that their biggest power is to transform the primary artifacts of your code before the rest of your program sees them. Rather being locked into a given syntax, the interpreter is open to extension, and you can negotiate with it for a new set of semantics.

In a way, by including decorators and allowing them to stand in for many of the use-cases where a new syntax might otherwise be desirable, the ECMAScript specification is firing itself from its primary responsibility of evolving the core language. What previously might have taken months or years, can now be done in a matter of minutes or hours; a huge roadblock gone that has historically stood in the way of progress.

For individual developers, decorators let us work with a lighter touch; applying just a little bit of new syntax to make big transformations, but all the while preserving the core intent and readability of our code.

I’m Charles Lowell (@cowboyd on twitter), and I build UI for a living at Frontside. If you enjoyed this, I’d love to hear from you. We're always hiring and looking for exciting new work

React Native and Chill: A tale of stupid made fast

Charles Lowell — Mon, 16 Jan 2017 12:00:00 GMT

The instant she told me, I knew with the feeling of faith that it was true. "Charles," she said gravely "You've got to build something stupid, and you've got to do it really soon."

Of course! Something stupid. Why didn't I think of that.

It was so obvious why not: I was too busy thinking about biznazz; too busy thinking about sales; too busy thinking about accounting. Too busy thinking about things like office furniture and hiring practices.

These are all things that are well and proper to think about... necessary even. They are the things that every voice I trust in my life admonishes me to concentrate on exclusively, my own loudest among them.

But listen too much to these voices and they'll have you piling the work-cart high with sundry projects and tasks. Now you spend every waking moment watching it teeter back and forth making sure it doesn't dump over and lose everything. Where is the inspiration in that?

Nope, time to set that cart down and do something stupid.

Luckily, fortune favors the stupid, and if you've set your mind to finding something idiotic to do, you're bound to succeed (based on the abundance of stupid things to do in the universe).

For me it happened the very next day.

A dumb idea

As a statement of fact rather than complaint, the end of 2016 and the beginning of 2017 has been a preternaturally stressful time. I know that this is what the internet has been droning on about lately, but in my case it's really true! I seriously don't remember a time in my life that was any trickier to navigate or that has required such sustained mental concentration.

Of course I have several coping mechanisms on hand, including Rye Whiskey, but one of the simplest and most effective ones that has come to my aid on more occasions that I can count is The Chillest Monkey. For me, I need only glance at his picture for a few moments before experiencing a physical sensation of relaxation.

His detached gaze sees through the edifice of anxieties I've built and twisted around myself. The low-frequency waves of tranquility emanating from his placid countenance dissolve and disippate it like a grease stain before a flow of warm detergent.

I was sharing this personal treasure with my co-workers when Rob exclaimed something to the effect of "We need to put this on the Apple TV!"

"Yes!" I thought. "Yes! we do!" We should have the power to behold the Chillest Monkey on an whim! And why stop with the TV? Shouldn't he be on the Watch and the iPhone as well?

I could almost feel the relief. "ZOMG!" I thought, "My life would be demonstrably better if this were the case."

Be bold, and mighty forces will come to your aid

The timing was fortititous: Apple TV support for React Native had just landed on master only a few short weeks ago. Clearly this was a sign from God that the moment to act was now. And indeed, with powerful allies like Robert Deluca, and Amazon Prime Now, how could I possibly fail?

So after building react-native from master to get tvOS support, and spending an afternoon with Rob hacking to make the chillest monkey layout properly on the Apple TV screen..... and yet a few more hours fiddling with XCode to install onto a physical device that Amazon had delivered just minutes before, the mountain was summited. His face expanded in animation to almost six feet across as the app opened up onto the projection screen. The feeling of tranquil gratification was palpable.

It's so comforting for me to know now that he's there; to know that all I have to do is pick up the Apple TV remote and conjure him with a gesture. But what kind of citizen of humanity would I be if I hoarded his boundless beneficence to myself?

The Chillest Monkey should be for anyone, and anybody that wants to include him in their apps and share his bounty with their users should be empowered to do so. That's why I created an extension that allows you to do just that: embed this most chill of primates into any application, on any device that React Native supports.

Hard made easy

I glossed over a lot of the yaks that needed to be shaved in order to achive the end result (and there were many). I learned about tvOS. I learned a lot about React Native and how to package components to share them between applications, but learning was never the point. And it didn't feel anything remotely like study. If anything, the sensation throughout was one of lightening my load, even as I was in the act of lugging a yard full of knowledge onto the ship.

And how about that?!? You trade in your worries and in exchange you receive new insights and skills. Now that seems like a good transaction if I ever heard of one... an arbitrage of the spirit you might say. Now remind me why I don't do this more often?

I don't think I'll need reminding of it in the future though. I also hope that whether it's using the Chillest Monkey in your own apps or something different altogether, you do something silly soon; something for which the payoff is dubious, but the downside doesn't exist.

I did, and it felt stupendous.

I’m Charles Lowell (@cowboyd on twitter), and I build UI for a living at The Frontside. If you enjoyed this, I’d love to hear from you.

Building infinite scroll in React Native

Robert DeLuca — Thu, 15 Dec 2016 12:00:00 GMT

In high school, I spent a long time trying to learn how to build iOS apps. After many months of attempting what seemed impossible at the time, I decided to throw in the towel. So I gave up and to went to design, which ultimately led to web work.

That desire to build native mobile apps never went away for me. But learning an entirely new language and ecosystem wasn't something that I could do in my spare time. So when React Native busted onto the scene in 2015 it caught my attention. Since it's written in JavaScript, React Native allows you (the JS dev) to use technologies you're already familiar with to build truly native apps.

Around the same time at The Frontside we started to write most of our libraries in pure JavaScript. This allowed us to wrap those libraries in whatever framework we happened to be working in at the time. That means we can take problems we've solved in Ember apps and use them in a React Native project. Which is exactly what we're going to do today!

Introduction

The purpose of this post is to build an infinite scroll component for React Native based on Impagination.js. We'll talk more about Impagination a little later but for now, just know Impagination is a lazy data layer for your paged records.

We're also going to lean on NativeBase for nice prestyled native looking components. If you're coming from the web think of NativeBase like Bootstrap. The main difference is the styling will be tailored to the OS running the app. So in iOS, you're going to get iOS styled elements and in Android, you'll get Android styled elements.

The API we will be using is a Rails app that returns paginated data which is generated from the faker gem. Impagination will work with any paginated API, so if you would like you can sub in your own API here.

Here's an iOS screenshot of what we're going to build:

You can also check out the repo for what we're going to build here. If you haven't set up React Native on your machine yet, you should take a moment and do that. The getting started docs are a great resource.

To sum up our goals for this post into a list:

Introduce Impagination
Create the React Native app
Create a component that will be shared in iOS and Android
Use NativeBase for styling
Create an Impagination dataset
Render that dataset to the app screen
Listen for scroll events to request new data

Hello Impagination

Before we start actually building our app I want to take second to talk about Impagination. Like I said earlier, Impagination is a lazy data layer for your paged records. All you provide Impagination is the logic to fetch a single page, plus how many records you want it to pre-fetch ahead of you. Impagination will handle the rest for you.

Impagination is built using an event-driven immutable style and has zero dependencies. That means you can use it anywhere that JS can run! From the server to the client, it doesn't matter as long as it's JS.

There are two required attributes for creating an Impagination dataset: fetch and pageSize.

fetch is a function that will tell Impagination how to go get the data you are requesting as you scroll through the infinite list.

pageSize is an integer that tells Impagination how many records are in each page.

Okay, that's enough of an introduction to Impagination. As we build our app I will stop and explain more of Impagination as we build the app.

Creating the app

Let's start creating our React Native app! I'm going to call this app robotImpagination since the gem generating our fake data uses a bunch of different robots for images. Feel free to name your app whatever you like.

react-native init robotImpagination

This will create a hello world react native app. If you cd into that directory and run react-native run-ios (or react-native run-android) it should start the React Native packager, build the iOS app, and launch the simulator.

Now that we have confirmed that the app will build, shut it all down. We need to install a couple dependencies to build our infinite scroll. Let's install NativeBase first: yarn add native-base. Then Impagination: yarn add impagination. You should now see both of those libraries in your package.json file.

No place like Home

Make your way to the index.ios.js (or index.android.js if you're doing android dev). You should see a couple imports and then a class that's extending Component:

export default class robotImpagination extends Component {
  render() {
    return (
      <View style={styles.container}>
        <Text style={styles.welcome}>
          Welcome to React Native!
        </Text>
        <Text style={styles.instructions}>
          To get started, edit index.android.js
        </Text>
        <Text style={styles.instructions}>
          Double tap R on your keyboard to reload,{'\n'}
          Shake or press menu button for dev menu
        </Text>
      </View>
    );
  }
}

Delete everything inside of the render method. We're going to replace that with a <Home /> component. This component will be used in both the index.ios.js & index.android.js files.

export default class robotImpagination extends Component {
  render() {
    return (
      <Home />
    );
  }
}

If you tried to run this right now, it wouldn't work. We still need to create that component and import it. For smaller projects, I like to put my components in a components folder. You can name it and place it where ever you would like. For me it's components in the root of the project: robotImpagination/components/Home.js

Inside of Home.js we should import React, the Text component from React Native, and create a component that returns "Hello!":

import React, { Component } from 'react';
import { Text } from 'react-native';

export default class Home extends Component {
  render() {
    return (
      <Text>Hello!</Text>
    );
  }
}

Now that we've actually created the <Home /> component we can import and use it in both of our index files (index.ios.js & index.android.js).

Your index files should look something like this now:

import React, { Component } from 'react';

import {
  AppRegistry,
} from 'react-native';

import Home from './components/Home';

export default class robotImpagination extends Component {
  render() {
    return (
      <Home />
    );
  }
}

AppRegistry.registerComponent('robotImpagination', () => robotImpagination);

Awesome! If you refresh your simulator you should now see a poorly styled "Hello!".

<Home /> will be the main component we will be working out of. This is so our code will run on both iOS & Android. Write once, compile to both!

Furnishing our Home

We now have an app that's uglier than what we started with. Don't fret, we're going to call on NativeBase to make everything look nice and clean. Inside of Home.js lets import four components:

// Native base for nice prestyled components
import {
  Header,
  Container,
  Title,
  Content,
} from 'native-base';

These components will provide the structure & styling going forward. I recommend taking 10 minutes to review the NativeBase docs. Now we can make our lonely "Hello!" text look much better:

// imports here

export default class Home extends Component {
  render() {
    return (
      <Container>
        <Header>
          <Title>Robot Impagination</Title>
        </Header>
        <Content>
          <Text>Hello!</Text>
        </Content>
      </Container>
    );
  }
}

This should give you a styled header with bold text inside of it and our "Hello!" text isn't in the very top left of the device. Nice!

Next, we should create at least one "card" where we will be rendering the content of each record into. Later on we will iterate over an array returning many of these, but for now, we're only going to create one card.

Thankfully we don't have to style this ourselves since NativeBase has a card component. We need to import two more components from NativeBase:

// Native base for nice prestyled components
import {
  Header,
  Container,
  Title,
  Content,
  Card,
  CardItem
} from 'native-base';

And additionally the Image component from react-native:

import {
  Image,
  Text,
} from 'react-native';

Now that we have access to these components lets fill it in:

// imports here

export default class Home extends Component {
  render() {
    return (
      <Container>
        <Header>
          <Title>Robot Impagination</Title>
        </Header>
        <Content>
          <Card style={{margin: 10}}>
            <CardItem>
              <Text>Hello!</Text>
            </CardItem>
            <CardItem>
              <Image style={{resizeMode: 'cover'}} source={{uri: "https://placekitten.com/640/440"}} />
            </CardItem>
            <CardItem>
              <Text>Item description</Text>
            </CardItem>
          </Card>
        </Content>
      </Container>
    );
  }
}

Our app should look something like this:

We have a nicely styled app with styled cards, lets start scrolling through paginated data!

Creating an Impagination dataset

The first thing we have to do is import Impagination into Home.js:

import Dataset from 'impagination';

Now create a method called setupImpagination inside of our Home component. setupImpagination is where we're going to create a new instance of Impagination, set the pageSize, and set the dataset on the components local state. Earlier I mentioned that there are two required params from Impagination to work. Let's start by filling in those two params.

export default class Home extends Component {
  setupImpagination() {
    let dataset = new Dataset({
      pageSize: 15,

      // Where to fetch the data from.
      fetch(pageOffset, pageSize, stats) {
        return fetch(`https://serene-beach-38011.herokuapp.com/api/faker?page=${pageOffset + 1}&per_page=${pageSize}`)
          .then(response => response.json())
          .catch((error) => {
            console.error(error);
          });
      }
    });
  }

  render() {
    // ...
  }
}

The fetch function is where Impagination will hit your API to get more pages. There are three arguments passed to the fetch function: pageOffset which is the current page it's going to fetch, pageSize which is a number of records in a page, and stats which can hold totalPages if your API supports it.

Note: you may need to add one to pageOffset since it's zero based.

This looks great! But we still have some work to do in order for it to work. We currently have no way of accessing the data that is emitted by Impagination.

Impagination has two different objects you will work with. One is state which holds all of the current records and the current state regarding that data. The other is the dataset which allows you to call Impagination methods like setReadOffset.

A new state is emitted every single time the data has changed. We can listen for these changes using Impagination's observe method:

export default class Home extends Component {
  setupImpagination() {
    let dataset = new Dataset({
      pageSize: 15,

      // Anytime there's a new state emitted, we want to set that on
      // the componets local state.
      observe: (datasetState) => {
        this.setState({datasetState});
      },

      // Where to fetch the data from.
      fetch(pageOffset, pageSize, stats) {
        return fetch(`https://serene-beach-38011.herokuapp.com/api/faker?page=${pageOffset + 1}&per_page=${pageSize}`)
          .then(response => response.json())
          .catch((error) => {
            console.error(error);
          });
      }
    });
  }

  render() {
    // ...
  }
}

Now that we're calling setState we have to create our state object in the component constructor:

export default class Home extends Component {
  constructor(props) {
    super(props);

    this.state = {
      dataset: null,
      datasetState: null,
    };
  }

  setupImpagination() { //... }
  render() { //... }
}

While we're here we'll also add dataset into our state object.

To pull all of this together and start fetching data we need to set the dataset on the components local state. Then we need to set the readOffset to record 0. This is so Impagination knows exactly what record you are on when scrolling through the list. If we get close to the end it will automatically fetch new records. You can read more about how this all works here.

The final look of our setupImpagination method:

export default class Home extends Component {
  constructor(props) { //... }

  setupImpagination() {
    let dataset = new Dataset({
      pageSize: 15,

      // Anytime there's a new state emitted, we want to set that on
      // the componets local state.
      observe: (datasetState) => {
        this.setState({datasetState});
      },

      // Where to fetch the data from.
      fetch(pageOffset, pageSize, stats) {
        return fetch(`https://serene-beach-38011.herokuapp.com/api/faker?page=${pageOffset + 1}&per_page=${pageSize}`)
          .then(response => response.json())
          .catch((error) => {
            console.error(error);
          });
      }
    });

    // Set the readOffset to the first record in the state
    dataset.setReadOffset(0);
    this.setState({dataset});
  }

  render() { //... }
}

Finally, as soon as the component starts to mount we want to setup our data store. In componentWillMount lets call setupImpagination:

export default class Home extends Component {
  constructor(props) { //... }

  setupImpagination() { //... }

  componentWillMount() {
    this.setupImpagination();
  }

  render() { //... }
}

Bam! Now when the app loads it will go off and fetch the first 15 records from the API and set it on the local state of the component. You can now access these records by doing this.state.datasetState.

Looping over the datasetState

With the styling and data now in place lets render it to the screen. this.state.datasetState is an array-like object. This means we can iterate over the state while still being able to access getters like this.state.datasetState.readOffset (which returns the current readOffset).

We're going to map over the Impagination state and return a card:

export default class Home extends Component {
  constructor(props) { //... }

  setupImpagination() { //... }

  componentWillMount() { //... }

  render() {
    return (
      <Container>
        <Header>
          <Title>Robot Impagination</Title>
        </Header>
        <Content>
          {this.state.datasetState.map(record => {
            return (
              <Card style={{margin: 10}}>
                <CardItem>
                  <Text>Hello!</Text>
                </CardItem>
                <CardItem>
                  <Image style={{resizeMode: 'cover'}} source={{uri: "https://placekitten.com/640/440"}} />
                </CardItem>
                <CardItem>
                  <Text>Item description</Text>
                </CardItem>
              </Card>
            );
          })}
        </Content>
      </Container>
    );
  }
}

If you refresh your simulator you should now see 15 kittens. But it's 15 cards with same text over and over again. We can start to pull information from each record we're iterating over:

export default class Home extends Component {
  constructor(props) { //... }

  setupImpagination() { //... }

  componentWillMount() { //... }

  render() {
    return (
      <Container>
        <Header>
          <Title>Robot Impagination</Title>
        </Header>
        <Content>
          {this.state.datasetState.map(record => {
            return (
              <Card style={{margin: 10}}>
                <CardItem>
                  <Text>{record.content.title}</Text>
                </CardItem>
                <CardItem>
                  <Image style={{resizeMode: 'cover'}} source={{uri: record.content.image}} />
                </CardItem>
                <CardItem>
                  <Text>{record.content.description}</Text>
                </CardItem>
              </Card>
            );
          })}
        </Content>
      </Container>
    );
  }
}

Refreshing your simulator should bring up an error message that says Cannot read property 'title' of null. This is because Impagination emits an array with 15 items as soon as it's instantiated. Each record in the array has five state properties:

isRequested
isSettled
isPending
isResolved
isRejected

With these properties, we can tell exactly what state each record is in. This allows us to display different UI for each individual record if we please. In this case, we're going to show a loading spinner if the record hasn't settled yet.

Import the spinner component from NativeBase:

// Native base for nice prestyled components
import {
  Header,
  Container,
  Title,
  Content,
  Card,
  CardItem,
  Spinner,
} from 'native-base';

Then add a conditional inside the map function that returns different JSX:

export default class Home extends Component {
  constructor(props) { //... }

  setupImpagination() { //... }

  componentWillMount() { //... }

  render() {
    return (
      <Container>
        <Header>
          <Title>Robot Impagination</Title>
        </Header>
        <Content>
          {this.state.datasetState.map(record => {
            if (!record.isSettled) {
              return <Spinner key={Math.random()}/>;
            }

            return (
              <Card style={styles.cardContainer}>
                <CardItem>
                  <Text>{record.content.title}</Text>
                </CardItem>
                <CardItem>
                  <Image style={{resizeMode: 'cover'}} source={{uri: record.content.image}} />
                </CardItem>
                <CardItem>
                  <Text>{record.content.description}</Text>
                </CardItem>
              </Card>
            );
          })}
        </Content>
      </Container>
    );
  }
}

This is great! It now will render each record with their own content. But it's pretty hard to read that render method, we should refactor the card into its own presentation component. Create another component in the components folder called RobotItem.js. Then copy and paste the card component code from Home.js:

import React, { Component } from 'react';
import {
  Text,
  Image,
} from 'react-native';

import {
  Card,
  CardItem,
} from 'native-base';

export default class RobotItem extends Component {
  constructor(props) {
    super(props);

    this.recordData = props.record.content;
  }

  render() {
    return (
      <Card style={{margin: 10}}>
        <CardItem>
          <Text>{this.recordData.title}</Text>
        </CardItem>
        <CardItem>
          <Image style={{resizeMode: 'cover'}} source={{uri: this.recordData.image}} />
        </CardItem>
        <CardItem>
          <Text>{this.recordData.description}</Text>
        </CardItem>
      </Card>
    );
  }
}

We also create a little shorthand in the constructor so we don't have to type this.props.record.content each time we have to access data.

There's still a little bit of refactoring left in Home.js. We're going to pull the map out of the render function and put it into its own method called renderItem.

export default class Home extends Component {
  constructor(props) { //... }

  setupImpagination() { //... }

  componentWillMount() { //... }

  renderItem() {
    return this.state.datasetState.map(record => {
      if (!record.isSettled) {
        return <Spinner key={Math.random()}/>;
      }

      return <RobotItem record={record} key={record.content.id} />;
    });
  }

  render() {
    return (
      <Container>
        <Header>
          <Title>Robot Impagination</Title>
        </Header>
        <Content>
          {this.renderItem()}
        </Content>
      </Container>
    );
  }
}

Don't forget to prune the imports we're no longer using at the top of Home.js.

Putting the "infinite" in scroll

We're so close! It's now rendering all 15 items in our first page but we're not able to scroll to the bottom and retrieve new records. Why? It's because as we scroll we have to set Impagination's readOffset. Basically, we have to tell Impagination which record is currently being viewed by the user.

As we progress through the list of records Impagination will fetch more pages if we're within the loadHorizon. By default, the loadHorizon is the same as the page size. This means Impagination is constantly keeping track of where we're at so it can smartly fetch new records as needed. If you up the loadHorizon to 30 it'll load even more pages ahead of the current scroll position.

In order to set the readOffset we're going to hook into the onScroll event on the <Content> component. The <Content> component descends from React Natives ScrollView component, so if you're not using NativeBase you can still use this same method.

Let's create a method called setCurrentReadOffset and then call that anytime the scroll event is called.

export default class Home extends Component {
  constructor(props) { //... }

  setupImpagination() { //... }

  componentWillMount() { //... }

  renderItem() { //... }

  setCurrentReadOffset = (event) => {
    // Log the current scroll position in the list in pixels
    console.log(event.nativeEvent.contentOffset.y);
  }

  render() {
    return (
      <Container>
        <Header>
          <Title>Robot Impagination</Title>
        </Header>
        <Content onScroll={this.setCurrentReadOffset}>
          {this.renderItem()}
        </Content>
      </Container>
    );
  }
}

If you refresh your simulator, enable remote JS debugging, scroll the list, and then look in the JS console you should see a bunch of logs with numbers. This is the current scroll position of the list in pixels. We're going to use contentOffset.y to help calculate what record we're seeing in the viewport.

Since this list is rendering items with the exact same height each time it's pretty easy for us to figure out what item is currently scrolled into view. You take the currentOffset.y and divide it by the items height. That's your current readOffset (aka the current record in view).

Finally, we should throttle the amount of times setCurrentReadOffset is called. As of right now it's called every single time there's a scroll event, which is extremely noisy. We'll cut this down by setting scrollEventThrottle to 300 (ms).

export default class Home extends Component {
  constructor(props) { //... }

  setupImpagination() { //... }

  componentWillMount() { //... }

  renderItem() { //... }

  setCurrentReadOffset = (event) => {
    let itemHeight = 402;
    let currentOffset = Math.floor(event.nativeEvent.contentOffset.y);
    let currentItemIndex = Math.ceil(currentOffset / itemHeight);

    this.state.dataset.setReadOffset(currentItemIndex);
  }

  render() {
    return (
      <Container>
        <Header>
          <Title>Robot Impagination</Title>
        </Header>
        <Content scrollEventThrottle={300} onScroll={this.setCurrentReadOffset}>
          {this.renderItem()}
        </Content>
      </Container>
    );
  }
}

Go ahead, refresh your simulator and scroll through that list! The API only has 100 records seeded to the DB so don't expect it to be truly infinite. Here's a GIF of what we've built together:

It's so beautiful.

One last optimization

Using a ScrollView for something that is truly infinate might not be the best idea but you can add one property to the ScrollView significantly cut the memory usage: removeClippedSubviews. Since NativeBase's Content component is backed by a ScrollView we can use this same property. removeClippedSubviews will take the off screen cards and remove them from the native backing superview.

export default class Home extends Component {
  constructor(props) { //... }

  setupImpagination() { //... }

  componentWillMount() { //... }

  renderItem() { //... }

  setCurrentReadOffset = (event) => { //... }

  render() {
    return (
      <Container>
        <Header>
          <Title>Robot Impagination</Title>
        </Header>
        <Content scrollEventThrottle={300} onScroll={this.setCurrentReadOffset} removeClippedSubviews={true}>
          {this.renderItem()}
        </Content>
      </Container>
    );
  }
}

Conclusion

First off, great job! You have just built infinite scroll in React Native from scratch, which isn't an easy task. In about an hour we were able to take a library that was originally written to solve a problem in Ember.js and apply it to a React Native Project.

I think this speaks volumes about the power of writing libraries in plain old JavaScript. Not only can this library be used in any type JavaScript project, it will have access to a broader community. This is because any JavaScript developer can jump in and contribute back to the project.

For me, this means I can finally realize my childhood dream of building an iOS app. I don't have to relearn an entire development ecosystem. Sure you'll have to learn new things about native development but the barrier to entry is tremendously lower.

Here's my hot take on where React Native fits into the JS ecosystem. React Native has hit the sweet spot that Cordova and PhoneGap has been trying to hit for years: you can write native apps in JavaScript with no performance implications. As we’ve seen, it’s easy to share the same JavaScript libraries from the web and node (if that’s your thing), to your native app. Now that’s amazing.

Web developers have been trying to recreate "that native experience" in the browser for years. But here's the thing: native is a moving target. So if you can't keep up, why not join them? But join them with your existing JavaScript knowledge. Sounds like the best of both worlds to me!

Once again if you would like to take a look at the completed app here's the GitHub repo. If you have any questions, comments, or feedback I'm always available on Twitter @robdel12

Image credit

The Ruby DCamp Experience

Stephanie Riera — Thu, 27 Oct 2016 12:00:00 GMT

Downfall of 2016

It's Fall season but with the current state of the world, it certainly feels like an extinction event is just around the corner. Clowns, Hurricane Matthew, and our country's infamous election has created a cold front of fear. Our social media feeds have been inundated in divisiveness and negativity. In the midst of this dark season, I had registered for Ruby DCamp and was ecstatic to disconnect from it all and attend my first DCamp experience.

Ruby DCamp (@ruby_dcamp) is a 3 day programming unconference in the forest where roughly 72 web developers from everywhere meet to bond over code and much more. Evan Light (@elight), a developer himself, founded the retreat dedicated to "knowledge, connection, and compassion" back in 2008. DCamp has been going strong ever since.

A time to retreat

It all takes place in Prince William Forest Park located one hour south of Washington D.C. tucked away in Virginia. The forest is part of the U.S. National Park Service with several campgrounds and trails to hike. If you're a nature enthusiast, you'll find the vegetation breathtaking and its lack of noise immediately calms the soul.

Attendees lodge in one of the several wooden cabins that were built in the 1930's and spend most of the day convened in the mess hall. The day's activities take place in the mess hall which also has a large kitchen. Everyone pitches in with cooking and cleaning, it's essentially communal living. There is no wifi and cellphone signal is spotty which makes you feel truly disconnected from the outside world.

The lone developer

In a competitive industry of "fake it til you make it" and "Whoa, you've never heard of Yarn?! Wow, ok, uhm...."- the act of speaking openly can feel awkward. People new to programming can feel intimidated attending hackathons or meetups and while they attempt to network, ultimately one feels like they're on this journey alone. Non junior developers can also relate to the isolation, simply ask them to write a blog post about the current state of JavaScript. The interwebz has no chill.

Learning to program is hard as it is and it doesn't need to be complicated by the inability to speak openly about fears of living up to expectations, handling imposter syndrome, or simply admitting what you know and don't know which is crucial for learning. DCamp actively fosters an environment that embraces vulnerability, which enables everyone to be sincere with one another. This habitat produces some deep conversations that make you realize, no, you're not alone. DCamp is essentially a sanctuary where you can exist and grow in an inclusive environment.

Code, dialogue, and playtime

The excitement is palpable on day one between the bustle of new attendees getting settled in and old friends coming together. Although DCamp is centered around the Ruby programming language, not all developers use Ruby as their primary language or even know it. The group is a diverse bunch with a spectrum of developers from junior to 20+ years experience. On day one, a coding challenge is explained and you are given 15 minutes to pair with a stranger and see how far you can get. The catch is, after the time is up, you have to clear your code, find a new partner and start again. The objective is not to finish the challenge but to see the different ways one can architect the solution.

The sessions were fun icebreakers and some of us found ourselves leaning on our JavaScript expertise or other languages to accomplish the same goal. Breaks, lunch, and dinner split the pairing sessions and before you know it, day one has passed. Day two and three are centered around unconference talks. That means the attendees themselves lead 45 minute discussions around topics of their choosing. Topics are posted on the main wall of the cabin, and you can cast 3 votes for your favorite talks.

The topics with the highest number of votes are selected and a schedule for the day is born. The talks are split equally between technical topics and nontechnical (people). Some of the technical talks focused on Ruby, Elixir, JavaScript, and sharing resources like websites, books, and podcasts. The nontechnical discussions varied from advice on pair programming, how to get your first programming job, to dealing with contempt culture.

It was humbling to sit in a circle of developers that admitted their struggle with accepting a compliment or shared the feeling of perpetual confusion and discomfort with programming. These sessions helped me see that we all share the same fears and that we all want a sense of belonging. Largely ignored, Ada Lovelace's published work fell on deaf ears until a century later when Turing rediscovered it. I sometimes wonder what technological advances we'd have today if both Lady Lovelace and Alan Turing had been accepted as people and had lived out their lives to old age.

If you feel accepted and safe to contribute in your team, you'll be less likely to avoid pushing code because you're embarrassed or fear someone making fun of your code or you as a developer. On the contrary, you should be sharing your code (no matter the quality), so that you can receive the feedback that will help you grow.

After dinner, the board and card games came out. There's nothing better than blowing off steam over some intense Secret Hitler and Werewolf games next to the fireplace. I enjoyed playing games but I ended up spending the last night from 8 pm til 3 am trying to solder a NodeMCU with built in Wifi that you can program in JavaScript, Python, or Lua. Mind you, I know nothing about hardware and have never soldered in my life.

It was amazing to see everyone sharing tools, collaborating, and teaching each other concepts they'd just learned. Shout out to Lance Gleason (@lgleasain) for purchasing & traveling with all the kits and tools for everyone to play with. Note to self: don't solder when it's 3 am, you'll find all the incorrect ways to attach wires in your exhausted stupor.

Mindfulness

One of my favorite aspects of the retreat was the focus on mindfulness. Everyday before starting the main activities, Evan would lead a short meditation. Pausing to take a moment to reflect, take a breath, and wish those around you well can sound corny but don't shoot it down if you haven't tried it. Studies have shown the benefits of meditation and I've seen first hand how a room of 72+ people can reach a state of relaxation you never knew was possible without music or drugs.

Spreading kindness, having empathy, and being self aware of our actions can make all the difference not only in your team but in how you interact with the world. If you're unaware of your facial expressions or tone of voice, you might not realize you're making someone inferior when you had no intention of doing so.

These moments of reflection helped me realize several things, one of which I'll share. An app essential for work communication like Slack actually hinders my productivity and I'd like to limit similar distractions in my life. I estimated (personally) 40% of conversations are of substance and the other 60% is just noise. I now check it when I get a ping but no longer go to it when I'm bored or stuck on a bug. I catch myself when I'm about to waste time going to Slack or Twitter, you'd be surprised to know it happens more often than you think.

During the end of my retreat I was full of these mini epiphanies and felt mentally refreshed. I gained new experiences, interpersonal skills, and technical knowledge. Best of all, I made friends that were strangers merely a couple days before. People were tweeting about returning to real life as if they'd just left Burning Man. DCamp has left a beautiful memory in my heart and I hope every developer gets the opportunity to experience it at least once.

I’m Stephanie Riera, I build web applications for a living at The Frontside, you can find me on Twitter (@stefriera). If you enjoyed this, I’d love to hear from you.

The Conjoined Triangles of Senior-Level Development

Brandon Hays — Thu, 07 Jul 2016 12:00:00 GMT

Stephen Tobolowsky expalining the cojoined triangles

My man Stephen Tobolowsky lays down the conjoined triangles.

"This actually makes me feel less confident about my role here. If we can’t define what we think senior is, how am I supposed to know if I’m working toward it?"

At Frontside, we gather every Tuesday afternoon for our company meeting, where we talk through our accomplishments and plans for the coming week.

In a recent meeting, we talked about our search for a senior developer to join the team, and you could see passions flare. No decision we can make impacts the company quite like bringing on a new team member, so naturally we all had strong opinions about the qualifications we were looking for.

However, not one person in the room could crystallize our thoughts on what qualifies as “senior” above a general gut feeling.

One person chimed in with the quote above, and I realized that we'd stumbled onto one of the biggest problems in our entire company: we had no idea how to define the role we're trying to hire for and grow our developers toward.

The problem of defining "Senior Developer"

For my part, I've come to distrust the term "Senior Developer", particularly after being given that title after 9 months of professional programming experience to fit into a salary band.

In fact, if you ask two experienced developers to specify their definition of "senior", I guarantee you will find serious conflicts between their answers.

The question of what qualifies as a “Senior Developer” is both context-dependent and so malleable that our industry squishes it into whatever shape fits their current needs.

Here are some actual, real-world definitions I've seen used to justify the label "Senior Developer"

15+ years of programming experience
2 years of programming experience and decent leverage
1 year of experience in a hot new 1-year-old framework
Having authored a technical book
Writing Comp Sci algorithms on a whiteboard
Writing an open source library used within the company

That's a pretty broad range of definitions. But lots of stuff in life is hard to define, so what's the problem?

Why bother defining it? What's wrong with a gut feeling?

In our meeting, when our team member pointed out their confusion, they were pointing out that we are hiring, firing, and promoting people based on criteria we can’t define or defend. They're right, that is bananas.

Worse, we were failing at our core mission of growing developers by not clearly marking the path we're trying to help them travel.

There's another, larger problem with "I'll know a senior developer when I see one": "Senior Developer" is a profoundly effective vector for bias.

"Senior Developer" as a way to enshrine bias

When we picture a wise senior developer, we each have our own experiences and preferences that color this picture in for us. That means the term is already spring-loaded with a bunch of our own baggage.

When we take a "gut feeling" sense of someone's seniority without specific criteria, there is basically no way to counteract our own biases, but we still make a judgement. It's completely possible for a person applying to multiple dev jobs to be evaluated as junior at one, mid-level at another, and even senior at another, with very little feedback as to why.

And as hiring managers, we all feel correct when making these judgements, even as we reach vastly different conclusions.

The result is a reinforcement of existing biases that keeps some people from progressing, while resulting in "title inflation" for others. Since existing biases in tech are strongly and undeniably tilted toward white males, the existing "gut feeling" system disproportionately harms women and people of color.

Why haven't we solved this?

This definition is an extremely hard problem, since it's heavily dependent on the context of the work environment. Most business owners are just making this stuff up as they go along, and the solution we tend to wind up with seems to be "good enough".

There's also no incentive, because defining criteria takes a lot of the "gut decision" power away from founders/owners and adds accountability. Who sits around asking for less power and more accountability for themselves?

Injecting accountability

I like accountability. I take comfort in it. I've learned that being held accountable for something is quite liberating. For instance, being clear about the criteria we use to hire someone leads to less regret than "going with my gut", which can be affected by everything from someone's ability to riff on Simpsons quotes with me to whether I skipped breakfast that day.

Accountability also opens the door for improvement. As the hiring manager, I'm responsible for fostering a safe, capable, happy, diverse team. Improving and working toward those goals can either happen by gut feelings and dumb luck, or we can define, measure, and hold ourselves accountable for iterating toward those goals.

Accountability moves us from being passengers to drivers of our own future.

It starts with responsibilities

The question becomes: How can we create and use quantifiable criteria for seniority without creating a deeply flawed, highly game-able system?

The only fair way I can think of to judge a candidate is asking a few key questions: What are this person's responsibilities? How likely are they to be able to fulfill them? How much help will they need?

We started by defining our environment. When we examine the key attributes of the environment at Frontside, it starts to become clear:

Being small means that people need to take the lead in several roles to solve problems from start to finish. There are no "cogs" in this machine.
We rely and build on the strength of our internal community and the larger communities we participate in, particularly in open source.
We have extremely ambitious technical goals and standards for the maintainability and usability of the code we write.

The resulting responsibilities are then pretty easy to determine:

Provide clear technical and project guidance for their teammates and our clients
Mentor, teach, and contribute internally and in larger programming communities
Ship software that is a joy to its users and to the developers who extend and maintain it

Together, these responsibilities form the basis of the criteria we use to assess seniority.

The Conjoined Triangles: The simple explanation

I recently had a chance to dig into the definition of "senior" at a number of companies, large and small, and came away with only one common denominator.

The simplest explanation of seniority across companies is this: How much direction will this person need, and how much will they be able to provide to others?

Graph that shows a linear relationship of what is provided by seniority

I stand by the Conjoined Triangles of Senior-Level Development as a nice idea, but like Action Jack’s "Conjoined Triangles of Success", it’s enough of an oversimplification to shed its intrinsic meaning.

Even with this understanding, the door is wide open to inject bias, miss important criteria, and overvalue higher-visibility things like public notoriety or the ability to speak computer science lingo.

In the meeting, a new framework emerges

A really cool thing happened during the heated meeting. As I was describing this simple "unified field theory", another employee put forward a new mental model to organize our thinking around this.

She described the framework we use to determine seniority at Frontside as a Venn diagram, the intersection of how well a person works independently and leads, how technically capable they are, and how well they connect and contribute to a larger community.

The Venn diagram: The more complex explanation

Venn diagram relating Connectendess, Technical Capability, and Leadership

Our evaluation of seniority does, in fact, roll up to the higher definition: “How much direction will this person need, and how much will they be able to provide to others?” But as our employees pointed out, when we stop there, there’s a ton of room for confusion.

So how can we decide how well a candidate is likely to perform their responsibilities? How do we anchor our evaluation criteria to something concrete, without killing it by turning it into a mathematical formula?

When we boiled down what we’re looking for, we came away with 12 traits that divide pretty cleanly along those three areas of responsibility: technical capability, leadership, and community.

I'll break the 12 traits down in detail in the next post, but for now, we'll just hit the high points.

The tracks

Technical capability: A person with high technical capability is technically curious, tackles problems without giving up, and produces solutions that less-experienced folks can use, maintain, and learn from.

Leadership: A person with leadership skills knows how to develop and follow a sense of purpose, in themselves and in others. They are willing to point out, own, and fix things that are broken about our company and in their own career tracks.

Community/Connectedness: A person with community skills has a sense of being part of a larger whole, a desire to contribute, a sense that the other people (i.e. coworkers, users, clients) are not simply characters in his or her own movie, but fully-realized individuals.

What about "culture fit"?

We initially almost called the Community & Connectedness track “culture fit”, but I have come to be highly suspicious that this term is a thought-terminating cliché. “Culture fit” is a junk drawer for all the stuff you want to see in a developer that you can’t otherwise define, and it’s an ideal hiding place for intrinsic biases.

When we defined the criteria that make Frontside’s culture unique, the common trait is the idea of connectedness as defined above.

Measuring seniority across 3 disparate tracks

Remember the three areas, technical capability, leadership, and community? Each of these tracks has a junior-to-senior path all its own.

It’s now common for a person to switch careers and come out of a code bootcamp with seniority in leadership and connectedness, while being junior in technical capability. Conversely, a skilled, formally-trained technician may be missing experience in both leadership and connectedness.

Very few qualify as senior in all three categories. In fact, it's unlikely a person will even be interested in becoming senior in all three categories. At Frontside, we define seniority as a blend of these tracks and work to help people level up where they feel like they want to improve.

Artifacts: The only measuring tape you've got

The measure of seniority in each track requires evidence. If you've done something, you likely have some kind of artifact as a result.

The 12 traits we'll discuss in the next post each come with concrete artifacts a candidate can point to to show that yes, they have developed those traits over time and with practice.

But taken as a whole, a couple of well-developed and deeply-practiced traits in an area are likely to qualify a person as senior in that area.

For instance, if a person can talk you through the code they've written across various programming languages, they probably peg the "curiosity" trait pretty high. Combine that with the rigor to write thorough, high-quality tests for the majority of their projects and hook it up to Continuous Integration, you're probably looking at someone we'd consider technically senior.

Or, if someone has a long history of mentoring others, organizing meetups, or creating tools that make others' lives easier, it's likely we'd consider them senior on the community track.

If a person has managed a few teams before, they have likely learned and developed processes to handle the very real challenges there. Combine that with the ability to get to the purpose or root of an issue, and you have someone we'd call senior on the leadership track.

How we define "Senior"

Our measure is that if someone is senior on the technical track and either the leadership or community track, they're senior to us. We can help level them up on the remaining track if they want.

If they're senior on the community and leadership side, we consider them senior if they are high-mid-level on the technical track.

As a real-life example: About a year ago, we hired a new employee as a junior developer, because we knew on the technical side they needed a ton of direction for at least the first six months.

At six months, they were solidly mid-level technically, and within their first year, by our estimation, they were already senior. I can say this with confidence because when they left, we realized that to fulfill their job functions, we'd need to hire a senior developer.

This happened because when they started with us, they were already senior on the community and leadership tracks, so all they needed was a boost on the technical side to perform the job of a senior dev on our team.

Technical skills aren't enough

For someone with technical skills but little experience in community or leadership, "You don't qualify as senior by our standards" is not music to their ears. But for the job responsibilities they'll have with us, we would consider that person mid-level until we can help strengthen one or both of the remaining tracks for them.

Many shops only measure along the technical axis, but it simply doesn't work for a company of our small size and collaborative work style. I also worry that those who only measure technical skill are buying into the dangerous "lone genius developer" myth, cheating the developer out of the serious boost they'd gain by adding leadership or community-oriented skills to their technical savvy.

Even in larger companies with more focused roles, I'd like to see a broader definition of "Senior Developer" that encompasses the code and non-code skills we use and that allow us to work effectively, particularly in teams and with real customers.

How long does it take to become senior?

Does “Senior developer” mean “X years of experience”? Well, I don't see it happening inside of 5 years for anyone. It would be difficult or impossible to develop the traits needed to reach senior-level in any of these tracks in less than several years, much less along multiple tracks.

But “5 years of experience” doesn’t necessarily mean “5 years of software development experience”. A person who has already climbed the Leadership and/or Connectedness tracks need only hook that up to their increasing technical skills to provide a tremendous amount of value as a software developer.

Our hiring "secret sauce" largely stems from the fact that it seems to take significantly less time for someone with leadership and community skills to develop technical skills than the other way around. I'm seeing a large number of people who graduated from code bootcamps 3 and even 2 years ago now handily and gracefully filling the role of senior developer.

There's more to discuss

This leaves many questions unanswered. What are the specific artifacts we use to evaluate skills and traits in the three areas? How can you evaluate for these before and during an interview? How do we tie these evaluations to something as concrete as someone's salary?

How does this framework apply to non-senior developers? How and when do developers advance? What's the difference between a Junior, Mid, and Senior? What exists between them? Are these terms devoid of meaning and in need of replacement?

Lastly, how does this framework apply (if it applies at all) to other companies with vastly different cultures and needs from Frontside?

In the next post, we'll dive into specifics to answer these.

Moving past "gut feelings"

Defining "senior" is an ongoing and surprisingly difficult process, but we do it because it's business-critical for us. Without a clear definition of “senior developer", we have no clear path for our own employees to get there. We have no concrete way to evaluate people joining the team, no way to hold ourselves accountable, and no way of improving the process.

As an industry, it's time to graduate from the “I’ll know it when I see it” definition toward something we can define and share. Let's bring open source thinking to how we hire and grow our people.

I hope to answer the major lingering questions in the next post. If you have questions or a different definition, you can find me on Twitter (@tehviking) or better yet, write a post with your own thoughts and I’ll link to it.

Accessibility & Why it Matters

Stephanie Riera — Thu, 19 May 2016 12:00:00 GMT

In honor of GAAD (Global Accessibility Awareness Day), I interviewed a developer who is active in the accessibility community to gain some insights on the importance of web accessibility.

Applying for jobs, registering your vehicle, selecting classes for the next school semester, and other tasks can become close to impossible if you have no access to the internet. Imagine for a second that you had no access to Google, Netflix, and dank memes. This sucks. Now let's imagine you couldn't use your computer's mouse or trackpad. How would you navigate the page or click on an image? Life would suck, royally. This is the reality for a population of internet users.

What is accessibility?

Accessibility refers to the design of products, devices, services, or environments for people who experience disabilities. Usability affects from the blind and color blind to the deaf and seizure prone. People in developing countries use the internet but may not necessarily know English. A site with poor usability makes it challenging for the elderly and those with motor skills disabilities to use.

([Image][3]: Blind students enjoying computer games at Techshare India 2012)

Federal websites must adhere to Section 508 which is a law that states Government sites must be accessible for people with disabilities. Some sites follow the ADA (Americans with Disabilities Act) guidelines loosely but the majority of websites could make a concerted effort to improve.

The dialogue is just not there.

And it's unfortunate. You'd think that with the current tech boom more developers and startups would pay attention to accessibility. The fact of the matter is, unless you have a disability or are directly affected by a family member with one, chances are you've never given it much thought. Even I didn't learn about web accessibility while I was studying web development.

([Image][6]: Google Trends graph of web accessibility and guidelines over time, captured May 16, 2016)

Despite the lack of awareness, this leaves room for opportunity. Accessibility applications can be expensive and have a steep learning curve. There is an untapped market with both native and web accessibility applications that have capacity for great improvement. The value you contribute through innovation and open source software is limitless.

Why does making my site accessible matter?

The UN Convention on the Rights of Persons with Disabilities recognizes access to information and communications technologies, including the Web, as a basic human right.

The internet has become more than humanity's encyclopedia. Cultural and social behaviors bubble out of the world wide web and become elements of today's society. The internet provides a medium for people with disabilities to interact. Everyone benefits when we support social inclusion for all. Tech is integrated in our culture and everyday life. Let’s try an experiment. For the next 30 seconds, navigate any webpage you want (this article can wait), and try navigating without a mouse. Hit tab and enter. See what it’s like to click links, fill in a form, whatever you like. Or better yet, if you have 3 or 4 minutes, turn on VoiceOver, and navigate with just the audio. Use a screen reader with voice recognition if you're feeling lucky.

By just taking those few seconds or couple of minutes, you can empathize with an entire world of users whose experience on the web is completely different from our own. And in the end, isn’t creating great experiences what software development is all about? This insight not only makes you a better developer but paves the way for building better UX. Beautifully designed sites avoid unnecessary noise and are navigable. Elegant applications incorporate architecture that flows and actions that are clear. Together we can make web accessible by default, the standard.

Alright, how do I start making my site accessible?

It's easier if you start putting thought into the site structure and picking colors from day one. Consider the flow of the site, would you be able to navigate through it if you were drunk? Test your colors with a contrast checker and start adding links to skip repetitive content. Imagine a screen reader describing the Facebook navigation every time you load the page. After the fifth time, you'd know the messages icon goes right after the friend requests.

Make sure you’re being semantic with your HTML tags. Use an actual <button> element and divert from doing this: <div class="button" {{action"showModal"}}>Open Modal</div> If you are in the practice of using <div> be cautious because the DOM doesn't recognize hierarchy. Divs and <span> should be used for styling, else a screen reader won't recognize a header and won't read it. If you must use a div as a button, please us the role attribute. This is a last resort.

WCAG 2.0 principle: Perceivable, Operable, Understandable or Robust

Never use a timeout because the user won't be able to get to the content in time and it turns into a videogame that you must master to get to the next page. Make use of checkboxes or radio buttons. Add labels for your forms and tables. An added benefit from making your site accessible is that it overlaps with other best practices such as mobile web design, search engine optimization (SEO), and usability.

Here are couple tools you can start playing with:

aXe is chrome extension that provides quick accessibility testing.
AChecker is available if you're not a fan of chrome extensions. Note: If you have a single page app it won't work.
ContrastChecker- does just that.
WAVE (Web Accessibility Evaluation Tool) also offers a Chrome extension.
AATT, Automated Accessibility Testing Tool helps you integrate accessibility testing into your existing automation test suite.

If you're seeking a wiki style run through of best practices, I highly recommend W3C (Web Accessibility Initiative), they have compiled strategies, guidelines, and resources on accessibility. Stanford University has generated SOAP (Stanford Online Accessibility Program) to provide resources for designers, developers and content creators. Hopefully these resources will make it easier to transition to building sites that are sophisticated, accessibility friendly and most importantly inclusive.

You tha real MVP

I'd like to thank Robert DeLuca for taking the time from his schedule to allow me to interview him on accessibility. Rob is a Frontend developer and a contributor to ember-a11y, an open source initiative to create Ember addons that help make accessibility default. He's also created awesome applications like DropKick, a plugin for creating beautiful, accessible, and painless custom dropdowns. You can find him on the Twitterverse (@robdel12) or talk accessibility on the Ember Slack (@robdel12).

I’m Stephanie Riera (@stefriera), and I build web applications for a living at The Frontside. If you enjoyed this, I'd love to hear from you! Shoot me a line at stephanie@frontside(dot)io.

Global Accessibility Awareness Day

Robert DeLuca — Tue, 17 May 2016 12:00:00 GMT

Thursday May 19th marks the fifth Global Accessibility Awareness Day. The goal for Global Accessibility Awareness Day (or GAAD for short) is to get people who work in technology to start talking, thinking, or learning about accessibility.

If you're not familiar, accessibility is about building solutions for those who have disabilities. For the web, it means considering those who are deaf, blind, colorblind, or have motor skill disabilities in your design and development.

A group of people that deeply care about accessibility and Ember.js have banded together. That group is called ember-a11y. Our goal is to make ember accessible by default as much as we can, and the first step in that direction is our ember-a11y addon that fills the gaps of the Ember router. Before this addon, those who rely on screen readers would never know the page updated when switching routes. Here is a quick video demonstrating this.

That's only the start of what we want to accomplish as a team. We're putting together a website that isolates and compiles some of the top ember addons on Ember Observer so you or assistive technology providers can test them. You can find the website here.

For GAAD 2016 we would love your help adding as many of the top component addons from ember observer as possible. To stop work duplication, add an issue BEFORE you start implementing the addon. That way, if someone else has already started work on the addon you wanted to implement, you would know.

If that doesn't strike your fancy and would like to contribute to addons, check out this spreadsheet. Here we've started a list of addons that need accessiblity work. The list can range anything from adding a role attribute to a div acting as a button, to pushing accessibility through for the wildly popular date picker component, Pikaday. To start working on something from the spreadsheet, place your name in the "assignee" column and highlight the row according to the legend at the bottom of the document. You can also add things to the spreadsheet you find from testing other addons!

You could also catch this Google IO event about Accessibility titled "Accessibility is My Favorite Part of the Platform". It is a great talk that we can learn a lot from.

If none of that looks interesting to you, at the very least spend 10-15 mins with a screen reader on any of your favorite sites. Try closing your eyes or dimming the screen and navigating only using the screen reader. This will help you understand how anyone who relies on a screen reader uses a website. This might be one of the most valuable things you do on GAAD.

If you're an OS X user, WebAIM has a excellent tutorial on how to use VoiceOver. Use CMD + F5 to toggle VoiceOver on or off. You can also toggle VoiceOver on your iPhone if ask Siri to turn it on or off.

If you're on Windows, you can use JAWS on a free trial or NVDA which is free. JAWS is the most popular screen reader currently. WebAIM also has a great JAWS tutorial and NVDA tutorial.

For me, this Global Accessibility Awareness Day will be sponsored by The Frontside. I'm going to try to give back to the community as much as possible. My day is going to be split into 3 parts:

Push accessibility through for Pikaday
Pairing hours [1-3pm]
Add template linting for common accessibility mistakes

I would like to pair with as many people as possible in 3 hours. Each pairing session will be 25 minutes long and we can work on anything accessibility-related you would like. This could be dropping the ember-a11y addon into one of your apps, solving an issue from the spreadsheet, or using a screen reader on websites. Sound interesting? Sign up here! We're going to be using Screenhero or Skype for screen sharing.

I'll be around all day to help. Feel free to reach out on twitter (@robdel12) or on Slack (robdel12)! For Slack you can snag me in the web-a11y Slack or the Ember community slack. What are you waiting for? Let's get out there and make the web a little more accessible! :)

So You Just Finished a Bootcamp, Now What?

Stephanie Riera — Tue, 17 May 2016 12:00:00 GMT

You took the plunge and here you are, a newly minted code bootcamper. You've endured the brain inundation of information and the inherent stress of not knowing what you're doing. Congratulations, you're here now and you're still asking yourself, "What do I do next?", I have no idea what I'm doing. You're here though, thanks Google.

As a developer, our job 90% of the time is to figure things out and create solutions using the resources we have available. You are a developer, you are just unemployed and finding a job is kind of the same process. It boils down to utilizing the available resources and creating your own solutions.

I've been in your shoes. I quit my secure job in a laboratory and decided to switch from Molecular Biology to Programming. I heard about bootcamps on NPR and felt like it was the best option given my student loan debt wasn't going to pay for my next degree. Drinking magma from the center of the Earth sounded better than another 4 years in college for a CS degree.

After finishing the bootcamp I wasn't sure where to go next. Most of the developer positions in my city had been occupied by the previous bootcamp cohorts and finding a job was proving to be more difficult than originally advertised. I have compiled all the lessons I've learned along the way and hope it helps you on your journey.

Expectations vs Reality

Bootcampers' expectation:

You graduated and by the end of the week you'll have your dream job in the bag. Better yet, your bootcamp will magically find the job for you. Cue turntable stopping abruptly. Some bootcamps may guarantee job placement but that doesn't translate to working in a desireable company or project. Alternatively, other bootcamps will refund you some portion of what you paid if you don't get a job after a period of time.

Technically, you are not guaranteed a job nor are you entitled to one. Some bootcamps help with the employment process by relaying available positions but you risk starting your career somewhere you shouldn't start at. You may very well find yourself unemployed 5 months after having graduated. I'll discuss this further in the next section.

Bootcampers' reality:

Finding a job is akin to speed dating- it takes effort. You put on your best outfit and go to the interview with the hope of impressing them while getting to feel them out. Don't forget, they're doing the same thing too. The relationship should always be mutually beneficial.

Every job will have have its own set responsibilities, company size, and culture. Before you start applying, sit down and think of what your ideal first job would look like. Do they participate in code reviews? Will they ask you to learn .NET and evolve into a customer support guru? Are they a TDD (Test Driven Development) shop or do they ship code lightening fast? Do you prefer working for a giant company or a startup?

What is important to you? Not sure yet? That's fine too because even if you land a not so great job, it's experience nonetheless and you'll discover what you like or don't like. It may take kissing a couple frogs.

Employer's reality:

Not everyone is a fan of bootcamp grads. Every bootcamp is different in how they present the curriculum and the scope varies greatly. You are not vetted and you lack the industry experience. Not all bootcamp graduates are good programmers and not all bootcamps prepare their students to be hireable. Combine those two and you have companies that have been burned from hiring bootcamp grads. If they consider hiring you then know that they are taking a risk on you. You are an investment and you are promising a return.

Some companies say they ain't got time for that and prefer hiring a mid or senior developer. Others offer internships or contract-to-hire gigs as a form of test trial before actually committing which can be advantageous for both parties. Then there are progressive companies that understand the value of building a diverse team of all backgrounds and levels of experience. These companies have mentorship and personal growth entwined in their company culture and they take it to heart despite the challenges.

Unemployment may happen, and yes it sucks.

If this happens, just power through. Even if money gets tight and you need a temporary job to pay the bills, remind yourself why this is temporary. Use your lunch breaks to catch up on podcasts or that JS book. Make sure any kind of free time is sacred and reserved to either studying or sleep.

If you have the luxury of staying at home for a couple months while you find a job, don't waste it. Seeing your reflection on the black screen between Netflix episodes sitting on the couch covered in cheeto crumbs at 2 pm is no bueno.

To avoid this, keep yourself accountable and productive by building your own 'work' schedule. This is what mine looked like until I was busy working 3 different side gigs:

	- Wake up between 8:30 - 9 am
	- Exercise
	- Breakfast & shower
	- Finding jobs, applying, & phone interviews
	- Lunch
		- Optional: Go for a walk or switch activities as a mental break
	- Work on projects, portfolio, OSS
	- Dinner
	- Go to meetups or keep coding if there are no meetups that evening
	- 10:30 pm bedtime (avoid 3 am interweb voids)

Whatever you do, don't stop working

Cast a wide net

There's more than just searching on Indeed. Freelancing builds your portfolio, you can find small projects on Upwork, Fiverr, and Craigslist. Leverage your connections and let them know you're in the market, they might know someone. Recruiters are a great asset because you don't need to pay for their services, they get a cut from the Company that hires you. If you do this, make sure you are specific on what you're searching for. Give the recruiter your website or GitHub profile so that they can get to know you. Do some research on nearby shops and check out their careers page, not all of them post on job sites. Don't forget to check the city government jobs and language specific job boards.

Nail your interview

Do your research on the company interviewing you, it shows you're interested in their product, values, and history. Get the basic interview questions down, practice makes perfect. I went to interviews for jobs I knew I'd turn down if I received an offer because I knew it was good practice.

Now for the sound of death: pair programming interview. This can be nerve wracking especially for a junior developer yet it can be a blessing in disguise. How? Well, this gives your prospective employer a litmus test of where you are as a developer. You can avoid unrealistic expectations and it's okay to admit when you don't understand a concept. Remember, your attitude under stressful situations says more than the code you're writing.

Last note, if you're a bootcamp grad, you probably became an expert in another industry. Don't neglect the fact that you had the courage to change career paths and the grit to learn a completely new discipline. You may not have all the experience right now but you have tenacity and other qualities that are equally valuable to an employer. You bring an unique experience and contribution to the table.

OSS

Open Source Software is valuable for everyone. When software is reviewed by hundreds of developers, it's hard for bugs to go unnoticed and the quality of the code is better than that of a 8 person dev team. Companies are jumping onboard the OSS train and nonprofits can use all the help they can get. As a developer you benefit from seeing how others create solutions and it's nice to contribute to a greater something. You can find the trending OSS projects on GitHub here and start pushing code.

Have an online presence

People are going to be doing their homework and looking you up. Let's face it, you're working in the tech world now, you should probably step your game up. Please do the following:

Use one (professional) email address for consistency.
Invest in a good headshot & use it across all your social media accounts. A white or brick background, good lighting and iPhone can do the trick.
Tidy up your resume and tailor it to the job you're applying for. Save yourself the trouble and have multiple resumes. (Ex- Retail resume, Developer resume, Marketing resume, etc)
Don't skip the cover letter. This is your opportunity to show off your writing skills and explain why you're a great hire they can't pass up. It also sets you apart from those candidates that aren't writing them.
You should have a LinkedIn and it should not be empty. Later it will come in handy to stay connected with the professionals you meet and you can search for jobs here too.
Push all your projects to GitHub.
Don't be lazy, write those ReadMe's. Without proper documentation, a user won't know how to use your app and has no where to go find answers. If you leave developers with just a directory structure, it's not only useless but a fool proof way of having them leave your repo.
Have your own domain where you link to your resume, GitHub, LinkedIn and other mediums like Twitter. This is a great place to exhibit your skills when asked about your experience.
Clean up your online profiles. Google yourself, remove embarrassing tweets and delete those questionable Myspace photos.

Be active in the community

Getting stuck in code at home is horrifying because it takes hours to get unstuck. Being around other developers can make a huge difference. Check Meetup.com for tech groups in your city and don't be shy. Meetups offer a safe environment to learn new topics through their lightning talks, the ability to present your projects, receive constructive feedback, and lose the speaking jitters. Meetups organize fun hackathons and it's a great place to get your networking on. Let me tell you when it comes to jobs, it's all about who you know.

Finding a coworking space in your area is paramount. You'll be informed of what is going on in your city's tech ecosystem, meet ambitious people, and even find out about positions with the local startups. Everyone here is busy hustling and simply being in that environment is better than trying to work from home or the loud Starbucks. These people like to work hard and play hard. The opportunity of making amazing friends along the way is real. Surround yourself with people that inspire you.

Never leave your home without your business cards and a positive attitude. If you don't have any, you can order some on Vistaprint or Moo. Your card should include your contact information, the title of the job you desire (ie- Backend Developer), and your GitHub account. Design your card to be an extension of yourself, people will remember you. There should be no room for missed connections, always do your due diligence and follow up with an email. A friendly smile can go a long way. Be the person that introduces themselves or starts a conversation with the person sitting by themselves.

Give first

Something I learned while working for Techstars was the value of giving first without the expectation of gaining anything in return. I can't stress how important and rewarding this is. You can be of service to others in a variety of ways:

Share advice in the form of tweets, a blog or a podcast. Sit down with someone who is interested in programming but is unsure. Sharing your experience even if you're not where you want to be yet can actually be inspiring to others.
Show what you learned at a meetup or at a brownbag lunch. Some schools host hackathons for children or have chapters of Girls Who Code and they're always in need of helping hands. You don't need to be a seasoned developer to explain how the DOM works or how CSS can add style to a page. When you support your local YMCA or code group for veterans, you are spreading knowledge while solidifying these concepts. The one who does the talking, does the learning.
Can't find any interesting meetups nearby? Create your own! I won't lie, it will take some planning and preparation on your end during the beginning but after that it gets easier especially after other organizers join. My bestfriend and I organized the first RailsGirls (a Ruby on Rails introductory hackathon for women of all ages) in San Antonio. We had a successful 30+ attendee turnout and I coached my own group without knowing a line of Ruby or the Rails framework.
Join a non-profit organization like Code for America where you can tackle civic problems for the greater good. You can always be of assistance remotely.

Offer your time free of charge. When you help out the local girls soccer team by designing their logo, you are freeing their budget to allocate funds elsewhere. You've become a contributing member of the community and when someone asks if anyone knows of a good designer, you have people that will vouch for you organically. Your name comes up everytime someone compliments their jersey.

Organic leads aside, let me tell you the most satisfying part. Knowing you've made a difference in someone's life. After the Rails Girls event, an older woman came and grabbed my hands to thank me for organizing the event. She said she never would've imagined that she'd be able to create an application from scratch. The majority of women who attended had never touched a line of code previously but left exhilarated and ready to continue learning because they knew it was achievable. I strongly recommend combining your passion with helping others because it's more fulfilling than you'd think.

Be kind, be patient

This journey can be a long one with unexpected twists and bumps on the road. I've given you all the pointers on how to hustle hard but I must also touch on personal wellbeing. I've asked you to be kind and of service to others but that should also include yourself. You might have moments of self doubt, frustration, anxiety, and I need you to know, I had those moments. If you know within yourself that this is what you want but you don't have the answers to everything else, that's okay. These difficult moments taught me how to be comfortable with being uncomfortable. These moments shape you. Have patience and trust in your struggle- even if those around you don't.

Circumvent outside noise. If you feel like your fire is being distinguished, listen to inspirational podcasts, read and watch autobiographies that amaze you. If you're feeling exhausted, listen to your body. Get a massage to relieve some tension. Go for a walk, watch a movie or go to bed early. Feed your body, mind, and soul. This trek is challenging but if you apply yourself, you'd be surprised at the things you can accomplish in a year. If there is a will, there is a way.

I’m Stephanie Riera (@stefriera on twitter), and I build web applications for a living at The Frontside. If you enjoyed this, I’d love to hear from you.

The Insider Track

Lydia Guarino — Fri, 18 Mar 2016 12:00:00 GMT

A guide to using kindness to elevate your conference experience.

Get your sticker collections, backup batteries and ironic nerd mashup t-shirts ready, because it's tech conference season!

We're gearing up for one of our favorite conferences of the year, and we've been spending a lot of time discussing what it means to have an amazing conference experience. We're incredibly stoked to see our favorite thought leaders, community gurus and internet celebs BLOW OUR MINDS with the new hotness and poignant epiphanies, but to properly set expectations, that is not what this article is about.

If you've got your ticket secured, you or your company has invested in the opportunity to create shared experiences with strangers from all over the world who happen to be passionate about the same things you are. You've pressed pause on your life and traveled far from home to connect, grow and recharge your techie batteries. You are here to be inspired to innovate and to uncover new opportunities for yourself or for your company.

Conferences are about community and community is all about participation. If you are attending a conference strictly for the speaker line up, you will likely have a better experience streaming the conference from home. You'll definitely get a better seat. But, if you want to fly home feeling like you just had the best three days of your life, after a whirlwind of fascinating interactions with people you once called strangers and now call friends, you have the conference spirit and it's time to supercharge your experience.

Let's back up a bit.

In a past life, before my development career, I was a sales rep for a large tech company. Once or twice a year, the company's 700 or so sales and vendor reps from all over the country would converge on Austin for a three day Carnaval of hyper-charged networking events that included extravagant parties, loud showboating at awards shows, loads of vendor swag, and lots and lots of people who lived life at a completely different pace than I did.

I spent a good amount of time at these events hiding in the restroom, recharging my introvert batteries so that I could put my extrovert mask back on. I'd then wade back into the crowd with a crazed smile painted on my face, feeling both overstimulated and exhausted at the same time. Sometimes I just stayed in my hotel room instead of going to the evening events.

It was only after several of these conferences that I finally figured out where my limits were, what my goals for the conference were, and how I could be empowered to avoid the things that exhausted me, so I could focus on the parts that were incredibly fun, inspiring and rewarding. I learned that I needed to be much more intentional with my time and energy.

Participation Over Observation

What worked for me was the decision to be a participant, not just an observer. When I was intentional about participating, I felt in control and was able to keep my energy up and my interactions positive. Each positive interaction fed into the next, inoculating me against the occasional not so awesome interaction. I left those events feeling like I was on top of the world. I learned there was a formula I could follow to dramatically enhance my experience and improve the opportunities I came away with after the conference.

I've followed this formula at tech meet-ups and conferences for the past three years. It's required very little tweaking. What was once a survival mechanism for me and has now evolved into an empowerment tool I use to forge strong connections with people who might be my next client, hire or boss. It's required very little tweaking because it's based on something pretty constant -- sales. And sales is based on humans.

You are a sales rep.

Face it, when you attend a conference, you are selling. This truth is scary for a lot of people in our industry. Sales sounds like something reserved for the pathologically extroverted among us. Luckily, it's a myth that you have to be incredibly outgoing to be a successful salesperson. It may help you get started, but it is not how you close a deal. You can't get someone to buy something by tricking them or by intimidating them into cracking open a checkbook. You've probably hung up on that sales person. They probably get hung up on a lot. And that's because sales is an exercise in investigation. It's the art of uncovering the intersection between someone else's needs and something you can offer. If you've ever had the opportunity to watch a skilled salesperson work their magic, you may have discovered that they do significantly less talking than you might expect. Most of their words are likely leaving their mouth in the form of questions.

While you may not be pushing products at a conference, you are peddling memorable experiences and positive impressions. Impressions that may lead to your next hire, your next client, or your next job opportunity. You're selling the idea that you are someone others would like to work with. You're selling the idea that you would be great to sit next to all day and someone who would be fun to collaborate with.

If this is not your first conference rodeo, you probably remember that guy that seemed like he knew and was adored by everyone at the conference. He was invited to do everything and was everywhere you went. While it may be true that this person naturally has some genetic defect that makes them fearless or has been sprinkled with some kind of unicorn magic, it is far more likely that they have just figured out the formula. It's a formula that will likely keep them gainfully employed for the rest of their working life, regardless of their objective level of technical skill, and it's surprisingly straightforward.

Focus your energy on making someone else's day better. Do that with every* person you meet.

This sounds fluffy, and it is. I've distilled it down into a mantra that I can repeat to myself when I walk into a big room filled with strangers. It's just a rephrasing of exactly what sales is: Uncover what someone else needs and figure out what you have to meet that need. The implementation of this mantra is what I call The Formula. Sometimes I use all of these tools and sometimes I just choose pieces for a specific setting. It took me a while to work up to using the entire list. I recommend trying out one or two that feel most comfortable at first and see how your experience changes. Then, try a few more.

The Formula

Part 1 - Setting the Stage

1. Wave hello.

The conference you are attending almost certainly has some social media channels associated with it. A couple days before a conference, start to make your presence known on social media by sending a message about how excited you are to be going to the conference. Thank the organizers or share helpful tips for getting to the venue or finding parking. Share a good place to eat that you discovered. Mention people you are excited to meet up with. Establish yourself as a friendly and welcoming person and people will be significantly more likely to reach out to you or be brave enough to walk up to you. Something that often gets overlooked is that your social media picture should be an actual recent picture of yourself. It's how people find you in person or friend you after a positive interaction.

2. Make an itinerary.

Come up with a list of things you want to do or see while you're at the conference. Pick out the talks you want to be sure to see, and pick some times to break off into the "Hallway Track." Some of the best interactions you can have at a conference take place in the quieter lulls when most people are watching talks. The Hallway Track offers a smaller pool of people to converse with. Find several restaurants, bars, after hours events, and recreation spots you'd like to visit. When people are huddled together after talks trying to decide on a place to go, offer an assertive choice. You'll get to do what you wanted to do, and people will appreciate that they didn't have to play the "I don't know, what do you want to do" game. It also gives you an easy way to invite the people you've had good interactions with to share specialized experiences with you.

3. Wear a costume.

This is an old sales trick. The Formula is about pushing past your comfort zone. Being outside your comfort zone can be incredibly exhausting. It's actually incredibly difficult for most humans not to dwell on what people think of you. You are intentionally putting yourself in a vulnerable position. I've had a lot of success with costumes. This doesn't mean you should parade about in a feather boa or a mask. It just has to be something that lets you be a little bit of someone else while you're trying to overcome being more of yourself. Sometimes, this is just a different hairstyle for me, or a new pair of shoes, or wearing something that reminds me of someone I love. It helps me get into my extrovert character, when my comfort zone would be to quietly sit in the back row.

Part 2 - Finding Positive Interactions

1. Ask, don't tell.

Small talk is the single hardest and most exhausting part of meeting new people. The trick to making it through this uncomfortable part of an interaction is to power through it with questions. Remind yourself that your goal is to figure out who this person is, what their goals are and what stands between them and achieving those goals. Your goal is not to impress them. You actually have very little control over that part. What you do have control over is your level of engagement with this new person. People are interested in people who are interested in them. People like people who like them. Asking someone about what's important to them is the best and fastest way to have a positive interaction. In a good interaction, that person will reciprocate and ask you about what's important to you.

Not everyone is going to be open to interacting with you, and that is totally okay. Your goal is to give them the opportunity to engage with you. If they don't reciprocate or you don't feel like it is going to be a positive interaction, it is perfectly okay to move on to speaking with someone else. Try not to take it personally. It's pretty difficult to tease out why someone might not want to talk to you. The best part about conferences is that there are a whole lot of people. The next person you speak to is probably going to be open to a conversation.

2. Be the includer.

This is my favorite tool. It's easier for some than others, and that's okay. Give it a try and see how it goes. If you've been in a particular industry for a while, it can be very tempting to spend most of your time at a conference with people you already know. You should definitely spend some time hanging out with the people you know and like, but you should also be actively figuring out how to connect those people with others you know -- or better yet, with nice people you just met. Sit down at a table with people you don't know. Invite someone that is sitting alone to sit with you. Invite people to go do something fun via social media. The more people you connect, the more those people will connect you with others. The best part is that if those people were nice, it’s likely that the people they introduce you to will also be nice.

Satistically, any time you are at an event with a lot of people, some minor subset of those people are not nice people. If someone engages you and you do not feel safe or they make you feel inferior or ashamed, quickly break that engagement and find someone else to talk to. If you see someone else having a bad interaction, offer them a safe escape and invite them to join you.

3. Play the yes game.

You've just a couple of days to soak up as many opportunities and experiences as possible. If you're lucky enough to be offered an invitation and you don't have an incredibly compelling reason to turn it down (unmovable conflict, you feel unsafe or you do not trust the person inviting you), say yes. It helps to think of this as a game. It will encourage you to try something you might otherwise bow out of. I'm a little scared of hack sessions. I play the yes game and I make myself go at conferences. It has always ended up being super fun. Bring a buddy if you are feeling timid! It's a perfect opportunity to be an includer. These invitations are almost always where the greatest opportunities are hiding.

Part 3 - Reinforcing Positive Interactions

1. Reward bravery.

Remind yourself that everyone you meet at a conference is being brave. If someone is brave enough to come talk to you, engage them! If you are lucky enough to get to talk to one of the speakers or organizers, thank them for the time and effort they poured into sharing something they care about with you. Publicly praising the brave people you've talked to on social media is a great way to reinforce the confidence needed to take risks and be brave.

2. Follow up.

Conferences are a whirlwind tour. Everyone you meet has met hundreds of people in a matter of days. If you uncovered an opportunity, don't wait to follow up. Before you go to sleep each night, queue up emails or social media touches to follow up with the people you met. Say thank you to the people that helped make your experience great. If someone offered you advice or encouraged you to try something new, reach back out to them to let them know you followed their advice and what the outcome was.

Community is about participation.

Conferences are about community, and community is all about participation. Participation happens on a spectrum. If you are attending your first conference or you're just working up to making the jump from spectator to participant, assess your limits. Don't go gangbusters on your first try. Pick a few strategies that sound like they might be in your wheelhouse and make that your first step. If those work well for you, try something you're a little more uncomfortable with. If all you can manage is to be kind to the people that talk to you, you've done your part to help build a positive community. If you can help one or two or twenty people have a wonderful conference experience, you'll walk away with some fantastic new contacts, some inspiring new ideas, enhanced confidence and perhaps a solid professional opportunity or two.

I’m Lydia Guarino (@lydiaguarino on twitter), and I build UI for a living at The Frontside. If you enjoyed this, I’d love to hear from you.

If you have a project you'd like to work with us on, get in touch. We're currently taking on new projects.

Functional Templating in Ember

Charles Lowell — Fri, 22 Jan 2016 12:00:00 GMT

TL;DR -- Avoid using "magically bound" internal component properties in your templates at all costs. Instead be explicit about only passing values around through actions and block params. This will engender a new level of breeziness to your UI.

Implementing a file upload in your web application is just the worst, am I right?

I mean, once you get your server set up, it’s on to choosing which off-the-shelf JavaScript Hairball ™ widget you’re going to use to actually walk the user through the upload process. And no matter which one you choose it never seems to fit quite right over the long-term does it? Let’s forget for a moment that they usually come with their own markup and style which is either difficult or impossible to replace with your own. As irritating as that is though, it’s something that we can perhaps live with, or at least wrestle into a submission hold. But where one-size-fits-all solutions really start to fall down is when you want to customize the actual file upload workflow. And even the simplest file upload workflows are involved aren’t they?

There’s not just the transfer of the bytes. First, there is the file selection process. If the file is an image, at the very least, you may want to show users a preview of that image. In some cases, you want to initiate the file upload immediately and without further action on the part of the user, but then again in others, you may need to stop and perform some intermediate work such as image resizing and cropping, and only then begin the transfer of the file to the server. If you’re uploading other MIME types like spreadsheets, there might be other intermediate preview-modify-and-confirm workflows altogether.

And so if you’re like me up until about a year ago, your experience with it was a constant and simmering frustration of whacking the square peg of an upload widget into the round hole of your application’s unique workflow, and caulking around the inevitable gouges and gaps with healthy dollops of JavaScript. In a word: Exhausting. Exhausting to stand up in the first place, and exhausting to maintain over the long haul.

So what would you say if I told you that with Ember components that are available on NPM right now you could build your own image upload widget, complete with preview and progress bars using nothing but about fifteen lines of Handlebars? Furthermore, what if I told you that you had complete freedom to use whatever markup and styling you saw fit, and beyond that, you could do it without a single line of JavaScript?

With the technique I’m going to outline, you’ll see that these are not outlandish claims, but rather the happy, daily reality of working with what I like to call functional templating. What makes a template functional? It's simple really. All it means is that the only way a name becomes bound to a value is via function application. That is to say, as a block parameter, or via a mut action.

A key construct underlying functional templating is the component that has little or no visual presence in terms of markup, but whose sole purpose is to manage a single piece of state and present it to a template.

I'm going to be using three functional components today to build our file uploader, the first of which is the file chooser. The file chooser is a component whose sole responsibility is to capture a FileList object and present it to the templating context. What happens to the files is not the concern of the file chooser however. That can be left up to other components later on down the road.

Let's see it in action! Click the "choose files" button below to select any number of files and have their metadata displayed in the area below the button.

In this example, the x-file-input's action emits a FileList object every time that the user selects a group of files. By binding (action (mut files)) to that action, we can inject that FileList into the handlebars scope, so that now it's available as a templating variable just the same as any other. Once it's in scope, the {{each}} iterates over the files and lists its metadata.

Notice how the x-file-input component in our chooser places very little restriction on the markup that activates the dialog. In this case we chose a button, but we could have made it a link, or a label, or picture of a cat.

Fig 1: Data flows from file input into template scope

Selecting the files and viewing all the metadata about them is great, but in our case, where we're headed is a full featured file upload complete with preview. In order to get there, we'll use two more functional components.

Preview All the Images

In order to preview our image, we'll need to display it in the browser. Luckily, the File API provides a createObjectURL method to help you do just that. Call it with a Blob object, and it hands you back a url that you can use for links, images, or any other place a url might be prone to go. When you're done with it, you call revokeObjectURL so that the browser knows it doesn't need to hang onto that Blob anymore.

And in order to present this url to our templating context? Yep, you guessed it. We'll use a functional component. We've written an object-url component which is responsible for managing an object url. It creates the url out of its blob attribute, and then revokes that url when either it receives a new blob attribute, or the component itself passes out of scope. Let's see it in action.

Note: In this example, the file input has been restricted to only image mime types by setting the accept attribute.

All we do is allow the file attribute to flow right into the object-url component which yields the url we need into the template to make the image preview.

Again, object-url has no markup of its own. Instead, its only job is to yield the model that makes the file preview markup possible. In this case, we bind the url value yielded by the x-object-url to the background-image property of a div and we're done.

Fig 2: object-url :: Blob -> String

Finally, we arrive at our destination: the actual upload. To do this, we use a functional component that executes a real life XMLHttpRequest!

This might seem totally nuts at first, but mostly because we’ve been trained over the years to think that a custom ajax request has always got to feel “heavy”. It turns out though that your plain vanilla, garden variety XHR is actually a well defined state machine, and as such, it maps pretty neatly onto a functional component.

We'll just let the file object flow into an x-xml-http-request component just like it did the x-object-url

Fig 3: x-xml-http-request :: File -> XHR

Again, the x-xml-http-request has absolutely no markup. Its only job is to continually yield a value into the template that represents the most up-to-date status of the request as it runs. Because of this, we can use it to very quickly model a progress bar! The only thing we need to do is leverage the xhr.upload.percentage property within the colorstops of a linear-gradient.

Heads Up! The following demo will actually upload images to posttestserver.com which is a real server on the real internet. Even though it’s just a bit bucket that periodically wipes all of its contents, you should still never upload anything sensitive to it.

And there you have it. A fairly decent file upload widget in about 15 lines of handlebars. It's worth noting though, that if you don't like the way I've done it here, you have a free hand to change the markup and workflow to suit your liking since the components I used don't prescribe any of it.

Dropping the F-Bomb

These demos were nearly trivial to put together because of the composability afforded by shifting to a more functional style. Values never just magically appear from some unknown location that could be inside your controller or your component. Instead, as in a functional language, variable names like xhr and url are only ever introduced as the formal parameters of a function. Because each component only interacts with its environment via its inputs and its outputs, it means that they can freely click together output-to-input; in as many ways as you can think of.

The first time this technique really came together for me it felt impossibly light-weight. It felt like I must be missing something. Could it really be that easy to upload a file? Could I really style it any way that I wanted? Did I really have control over every aspect of the entire workflow from start to finish? And could I really do all of that while providing little or no custom JavaScript? The answer to all those questions turned out to be yes, and the resulting feeling was spectacular.

I’m Charles Lowell (@cowboyd on twitter), and I build UI for a living at The Frontside. If you enjoyed this, I’d love to hear from you.

Also, If you'd like to work with our team doing stuff like this, then please get in touch. We're hiring.

X-Select Status Update

Robert DeLuca — Fri, 14 Aug 2015 12:00:00 GMT

At The Frontside, we love open source, and maintaining OSS projects is a high priority.

emberx-select has grown in popularity. Even the ember deprecation guide recommends it. We are aiming to keep this component up to date with the latest versions of Ember.

Before Glimmer (1.13) compatibility, we are making a release for Ember 1.12 or below.

What's new in 1.1.3?

Title attr bound to an option or select. PR
Add optionValuePath PR
Fixed blockless form PR
Default tabindex to 0 to make it tabable. PR
Introduce registerSelectHelper PR
Improve test helper PR
Improved testing coverage

See the diff here.

Thank you to everyone that has helped by submitting issues and pull requests! 1.1.3 will be the last version released that's compatible with Ember 1.12 or below.

What's new in 2.0?

In x-select 2.0 we dropped support for the blockless form. While it is useful for transitioning from Ember.select, we strongly prefer to standardize writing selects in Ember as in HTML. We no longer bundle x-select with the blockless form. To continue using blockless x-select, use this addon it from x-select.

x-select 2.0 arrays are now immutable. Every user selection creates a new state of the selected content. Previously, multi-selects required updating content as a user interacts with the select component. This was unstable especially with async ember data relationships.

Expect Glimmer/Ember 2.0 compatability in x-select 2.0 thanks to this PR from James Rosen!

Known Glimmer issue with x-select

To use x-select with Glimmer (Ember 1.13.x) requires 1.13.4+. Anything below 1.13.4 will throw

Error: Assertion Failed: x-option component declared without enclosing x-select

because non-dirty component child views are not getting the correct parentView. You can see the related ticket here.

We also applied a small hack to work around form attribute bindings which are broken in Ember 1.12-1.13.2 and fixed in Ember 1.13.3. Ember data bindings are lost, so the parent form should not change. If data bindings are still necessary, call rerender() on the component after the data change.

Once again, thanks to all the contributors that helped make the x-select component better for everyone!

Using x-select-blockless

To use x-select-blockless you'll need to install both x-select-blockless and x-select ember addons. This is because we rely on x-select.

Compatibility Table

x-select	Ember
v1.1.4 and before	1.12 or lower
v2.x.x and after	1.13.4 or higher

Programming in the Wild West

Brandon Hays — Fri, 16 May 2014 12:00:00 GMT

Much like revisiting a neighborhood from my youth, when I reflect on my scant few years in software I'm astonished at how much has changed.

This might have also struck you at some point. Perhaps you were even a pioneer in your community's wild west days, huddling in log cabins of your own construction, only to look around now and see yoga studios and gourmet cupcake shops.

The pattern of how and why this happens is actually related to the way the "red-green-refactor" loop in test-first development helps match our work to our brains. But more importantly, it's reflective of how we can adapt and help our communities as they mature.

First, a journey into the human brain

Studies are revealing that during complex tasks, each of us switches between "open mode", primarily associated with free-form experimentation, and "closed mode", associated with focused execution.

Sarah Mei spoke about this last year at LoneStarRuby, and compared the TDD cycle to moving between these modes. (Starting at 12:20)

The general idea is that at different times, people are better at thinking about the big picture, or zooming in on the details, but not both simultaneously.

This sounds like a simple concept, even obvious, but we ignore it so much that when we actually put the idea into practice, it can be life-altering.

Test-first and open & closed modes

The TDD cycle

First, I should clarify that this is representative of how I've learned to define and practice test-first development, and others may have different definitions or methods.

For me, this pattern has helped me leverage the strengths of open/closed modes in my brain, because it compartmentalizes times for experimentation and then execution. Much of my own "open mode" experimentation happens via a practice that I don't often hear discussed in TDD circles: spiking code.

The "spike" is where you throw away best practices, testing, and most forms of discipline in favor of discovery. You get out the machete and hack a meandering path from Point A to Point B.

Spiking is the act of figuring out what question you need to be asking. The "red" step is asking the question, then the rest of the TDD process is about answering it. To me, the spiking phase is an essential part of the "open mode" quest to ask the right questions.

The only rule of the spike is that it must be thrown away. You now know where Point B is, and the second trip through using discipline and techniques like TDD will reveal a much straighter path.

But it's an extremely bitter pill: throwing away functioning code is one of the most ego-shattering, counterintuitive things I do as a developer. However, I've learned it's typically faster to subjugate my ego and just delete the code.

The spike clears the "fog of war", shedding light on real-life obstacles in my initial design. Their concrete result is a set of high-level integration tests, from which I can drive out a new implementation in a more direct, point-A-to-point-B way, without having to machete-hack through the jungle.

How this relates to software communities

Software communities, unsurprisingly, behave much like the humans who comprise them.

Yehuda Katz gave a landmark talk at RailsConf, inspiring this post in the first place. He talks about when it's time to move from experimentation mode to shared solutions mode in your community. (Starting at 13:25)

The talk covers several topics, but the main point to me was that software communities have trouble toggling between experimentation ("open") mode and shared solutions ("closed") mode.

To me, this looks a lot like the test-first cycle above:

The community cycle

Early in a community's maturity, it may be time for a bazaar of different solutions while everyone is simply learning the right questions to ask. What does an authentication solution look like? What about persistence? Security? Code sharing? Scalability? Deployment?

Once a community settles on those few key questions and sees some possible solutions to each, the time comes to get behind existing solutions and improve them. Often, these early projects aren't extremely pleasant to use or well-factored, but can serve to clear that "fog of war" inherent to a new domain. But once the fog is cleared, one or two solutions to a given problem usually gain traction within a community.

"Worse is Better"

When it feels like a community is rallying around a shared set of abstractions, it may be time to delclare it a platform and move from experimentation mode to reaching an agreement. Examples of this in the past include the merging of rival Ruby frameworks Merb and Rails, or the Promises/A+ spec in JavaScript.

In order for this to work, people in a community must be willing to put their energy and focus behind a set of solutions that they don't see as ideal. Maybe they even think they suck. Yehuda used the phrase "worse is better" to describe this. The concept is generally that after the time of experimentation, a community is better served by embracing a comparatively crummy solution and improving it.

There's friction when this change starts happening. Some people feel left behind by the community they helped build. Others see the flaws in existing solutions and want to take their best shot at it, though the time for experimentation has passed. Many may feel that no shared solution could possibly approximate their unique needs.

The good news is that after agreeing on a solution, even if it's a sub-optimal, you can improve the new shared solution, and without much fanfare, there's a new platform for everyone to build upon.

Platforms and scaffolding

Each new layer adds a level of abstraction, so you need to have confidence in the layers beneath you. In software projects, this confidence comes from good tests. In communities, the confidence tends to come from seeing a project in broad use.

We don't spend a ton of time wondering if UNIX is going to crumble beneath us when we're building web applications on top of it, or that the V8 engine in Chrome is just going to stop evaluating JavaScript.

At some point, we declare these to be stable platforms and simply build on top of them without constantly running down to the basement to make sure the furnace didn't explode.

Programming in the Wild West

When I was first learning to program, I was taught a term for people who refuse to leave the "spike" phase of programming. They're called "cowboy coders", often disparagingly. Many of my favorite programmers work in this style, so I use the term hoping it implies no judgment on my part.

Communities have an equivalent type, the "pioneer" who prefers to live in the experimentation phase. Pioneers would rather invent a persistence library or templating engine than spend their day building apps in an existing framework.

When you are in the Wild West, you are, by nature of your presence, a unique and special snowflake. You're the pioneer. You could be the first doctor, the first newspaper editor, or the first blacksmith. All the problems you encounter are by definition novel, and you must survive by your own ingenuity.

Pioneers have much to offer in the "wild west" phase of a developer community. There are few tools, no consensus, and little infrastructure, so there's ample opportunity to contribute in a significant way.

The Rust community seems like a good example of a current-day wild west, leaving many questions yet to be asked, much less answered.

The taming of the West

But after a while, the gold rush has passed, and people have set up Costcos and Pilates studios. The experimentation phase is over, and people have decided on a platform.

Programmers using Rails might remember this starting around Rails 3.0. After that, the big questions were answered, and a few solutions emerged that have evolved into what are essentially just two technology stacks.

The problem occurs when remaining pioneers still want to build a hospital, but there are now plenty of decent hospitals. The community has matured to where offering a lot of alternative solutions start doing more harm than good, creating confusion and decision fatigue.

How do you know a community is reaching the end of the experimentation phase?

You see articles saying "please no more {{x technology}} testing libraries/package managers/persistence frameworks"
Consultants specializing in {{x technology}} proliferate
Both traditional and hacker schools start teaching {{x technology}}
You see job postings at Fortune 500 companies for {{x technology}}
{{x technology}} is on the cover of magazines targeted at CEO types

The bitter pill

Remember the bitter pill in TDD, throwing away the spike to embrace discipline and move into closed mode?

As a software community's experimentation phase wraps up and moves into shared solutions, developers have a bitter, ego-shattering pill to swallow, as well. It's the admission that you are not a special snowflake.

If you want to continue working in the community you helped grow, you suddenly have to contend with the fact that many other people moved in and share your same set of problems. It can be brutal to face the idea that a community stands to get greater benefit if you now pour your efforts into shared solutions than into a bespoke, one-of-a-kind tool.

Go West!

Some people have an insatiable desire to pioneer. They are never satisfied with the state of things and want to see the best solution prevail. As they see fences erected and freeways built, they can feel frustrated, as if their freedom is being taken away.

A Rails developer may be tempted to create a better persistence framework, but the community has largely agreed on ActiveRecord and is actively working to make it better.

So what's a pioneering spirit to do? That's the tough part. You can pioneer in ever-smaller niches within your community, swallow the bitter pill of not being a unique snowflake and work on the "worse" solutions, or head west in search of a less mature community in need of doctors and blacksmiths.

You may even find it rewarding to play around with all three options. And the best part is, there is an infinite frontier for programmers. You can always go west, there is always something new to explore if your desire is to chase the sunset into the unknown.

Understanding open/closed modes is good for your software

I am insatiably curious as to why things work, and I'm so glad that Sarah touched on exactly why TDD works in her talk.

I started as a "cowboy coder" and resisted testing for a long time, particularly TDD. It feels restrictive to have to compartmentalize. I already know how to make something work. But it doesn't take me much time to realize how expensive it is to try and do both modes at once, the complexity catches up with me, and I end up wishing I could throw the code in the garbage instead of maintain it.

In TDD, you're dividing the incredibly complex task of writing software along the brain's natural modes. In a designated period of "open mode" thinking, you start to ask the right questions, giving you a concrete result: failing test cases.

You then have a natural break into closed-mode, solution-oriented work, and back again. You let the tests act as a sort of bookmark for your thinking as you toggle between modes, keeping you in context and making you more productive.

Let's look again at the circular diagrams above. From that perspective, they look positively Sisyphean and don't tell the whole story. In reality, they are the means by which these towering platforms are built, but you have to view the circle from a third dimension to see it:

The TDD cycle, from the side

With good tests, every trip around the loop provides more capabilities on top of a trustworthy codebase. You can keep building without fear of your codebase collapsing under the weight of its own complexity.

Understanding open/closed modes is good for communities

In Yehuda's talk, he plays a couple of videos of Steve Jobs talking about using frameworks to provide a solid platform, so you can build your 5 stories starting on the 25th floor, rather than the 5th floor.

In open source software communities, we can't simply buy into these platforms, we have to agree on them. This happens in the same open/closed cycle: we discover questions by experimenting in open mode, then diving into the closed-mode work of converging on a solution and improving it.

The benefit is an exponential gain in productivity because we aren't exhausting ourselves by keeping 25 floors of scaffolding from collapsing, and instead focusing only on the few flights that comprise our software's unique needs.

At each turn of the loop, a community is establishing a platform for the next go-round. Open source has done a phenomenal job of this over the last 30 years, and the cumulative effect is simply staggering.

The pain of moving into "closed mode"

When Yehuda outlined the difficulty in moving from "experimentation" mode to "shared solution" mode, I felt as if a gong went off in my head, and I suddenly understood much of the friction in the maturing JavaScript community.

I'd contend that we're reaching the end of the JavaScript community's experimentation phase, particularly as it pertains to building client-side applications. We're starting to see people setting up Pilates studios and Costcos. It's entirely possible we'll be talking about ES6 as a watershed moment, the way Rails developers sometimes do about the shift to Rails 3.

This can be rough on the pioneers who got us here. It can also feel constrictive to those who want to build a better version of an existing solution. It's not easy for me, either. I want great tools, not mediocre ones. But if we want the benefits of building on top of platforms instead of our own rickety scaffolding, our responsibility is to embrace imperfect solutions as they get traction, and then improve them.

I'm glad to have the opportunity to participate in the JavaScript community as it starts to converge on "platform-level" solutions. If we'll work together and embrace that we're not "unique and special snowflakes", we can focus on the few floors that actually make our software unique and valuable to other people, so the next generation of developers can start building on the 1000th floor.

Ember and the future of the web

Brandon Hays — Mon, 24 Feb 2014 12:00:00 GMT

If you know me, you know it's easy to get me talking about Ember.js. A year ago, I was deeply skeptical of developing client-side applications. Now, after spending a year shipping production applications in Ember, I believe that client-side apps represent the same generational leap that database-backed apps with Rails offered circa 2006.

Despite the title of this article, I can't actually predict the future, but I can bet on it. And Charles and I are betting The Frontside's future on the fact that Ember provides a foundation for creating amazing, next-generation user experiences for the web.

What server-side MVC does well (and doesn't)

Server-side MVC frameworks rely on the classic Apple philosophy of optimizing for the 80% use case. These frameworks excel at taking a database, exposing CRUD (Create, Read, Update, Destroy) actions via HTTP, and building form-based UIs to work with these records. CRUD-oriented frameworks make it easy to build applications like blogs and contact management software.

What server-side MVC frameworks are not particularly well-suited to are user-driven, stateful UIs like you'd see in desktop apps like Word, Excel, or iTunes.

Rails, as a mature framework, has evolved a set of standard solutions to these types of problems. Generally, they involve minimizing the amount of JavaScript needed and placing as much of the burden on the server as possible.

Server-side workarounds

I've built these many times "the Rails way", and I can viscerally feel myself cross the boundaries of the things Rails does well. I might start by creating a template called create.js.erb to be triggered from new.html.erb.

$('#tag_form').html('<%=j render("form") %>');
$('#tags').html('<%=j render(@el.all_tags) %>');
initSelect2($("#tags"));

But what's actually happening here is not simple (and has security implications that bear keeping in mind):

(Credit: Egor Homakov's blog)

And that's a greatly simplified example. The fact is that if you're building stateful client-side interactions, you're already managing a lot of JavaScript, but your code is probably scattered across jQuery plugins, server-side rendered JavaScript, and custom JS in your assets.

These patterns quickly become difficult to maintain, and the pain encourages developers to steer clear of complex interactions and focus on the Rails happy path.

And this isn't necessarily a bad thing. We love Rails for making those tradeoffs on our behalf, so we can focus on the 80% case.

But I have two cautions about this kind of pragmatism:

Trying to stretch the 80% solution into the other 20% almost always results in pain.
Keep your eye on the 20%, because it may just be the next 80% use case.

Two cases for using Ember

Ember, to me, indicates a seismic shift in the way web applications are created. And not because Ember itself is so great (although I do like it a lot).

Rather, it's that Ember makes it fun to splash around in the other 20% and build next-generation applications.

I'll make two cases for Ember and then explain how I see it as a representation of the web of the future.

Case 1: MVC/MVC is a feature, not a bug

I'm a classic pragmatist. I like to learn things with an eye toward solving problems. So the concept of an MVC framework on top of an MVC framework struck me as odious. But since then, I've come to see it as incredibly freeing.

I never realized how much of my thinking was shaped by the limitations of the tools I used until exploring something with a totally different set of capabilities and constraints.

Let's look at the original Rails app, Basecamp:

Basecamp

Yes, the screenshot is outdated. This is not meant as a knock against Basecamp, which has been since rewritten, but rather an example of how most of us write our Rails apps. Server-side MVC frameworks lay a yellow brick road right to that interface. It's certainly representative of the majority of my vanilla Rails applications.

A few months ago, we started using the chat service Slack. I feel comfortable saying that it is a good example of web applications in 2014: it is so deeply interactive and so native-feeling that it nearly made me choke on my Cheerios.

See if you can spot a direct mapping to database rows in Slack:

Slack

I sure couldn't pin it down. In fact, after downloading the standalone Slack app for OS X, I was sure that it was a native app until I right-clicked and saw "inspect element" in the menu. It's not an Ember app, but I can easily see how it would be done.

Client-side applications basically require you to saw your applications in half. Handle your persistence layer however you want, just expose APIs to create, update, fetch, or delete records (a task to which Rails is particularly well-suited).

The other half of your application is now 100% tailored to the user interface you want to build. You find yourself using the language and workflow of your users, rather than wiggling around within the constraints of your database tables and columns.

All of this is wonderful, but represents any client-side MVC, rather than being unique to Ember.

Case 2: You Need Models, You Just Might Not Know it Yet

Ember has a lot of strong suits, but one of the strongest is its rock-solid model layer. There aren't many hard UI problems that can't be made much simpler by using models to back your interactions.

Back to my personal journey: After a while, I abandoned RJS and started using plain JavaScript and jQuery to manage complex interactions.

My new AJAX-y workflow went like this:

Let the server render stuff to the page
Set up a controller to handle AJAX requests from the UI
Manipulate the DOM using a jQuery plugin
Traverse the DOM again to see what changed
Serialize changes from the DOM and send via AJAX
The server returns an HTML string that is stuffed back into the DOM
Repeat steps 3-6
Spend days troubleshooting inconsistent UI results

I'll spare you the gory details, but I've written a ton of code that looks like this:

// ZOMG remember to clean up row position or everything breaks
var renumberRows = function(){
  $("#sortable .content_item:visible").each(function(index){
    $(this).find(".row_num").html(index + 1);
  });
};

Not only is this style not simple, it's quite difficult to test. My application is flying blind on steps 3-5, which, as it turns out, is the absolute core of the user experience.

With Ember, these DOM manipulations are side effects of working with models. A user doesn't know it, but they're pushing actual data around, toggling boolean attributes, and moving full-blown objects between lists.

Here's an Ember computed property to illustrate:

// Model, you worry about position and I'll go get a soda
sortedRows: function() {
  this.get("rows").sortBy("position");
}.property("rows")

I can't show actual comparison code to the renumberRows() function because in Ember, there is none. Any changes rendered on the screen are the automatic result of changes to model data. With Ember's router, even your URL is rooted in this model data.

Think about that for a second. Imagine never writing another line of jQuery code that second-guesses or cleans up inconsistencies from your DOM.

Ember's object model, its system of observers and computed properties, and the way these are bound to the DOM are second to none. Its model layer isn't bolted on or mixed in, it's the foundation of the framework itself, and is absolutely rock solid.

Looking out for the next 80%

The book The Innovator's Dilemma is a fantastic read, and, I believe, applicable to the state of MVC frameworks.

The same pragmatism that allows us to get things done can actually become toxic after a significant degree of success. Almost by default, people tend to try and extract the maximum value out of existing solutions. They don't notice as the ground shifts beneath them, and are often surprised to find that their old, reliable solution has been disrupted.

By nearly all accounts, when Rails entered the scene it acted as this disruptive force. As anyone trying to bring Rails to work projects before 2010 will tell you, it was a tough sell, often being shouted down for not being "enterprise grade".

To me, the most powerful feature of Rails was that it took things that were excruciatingly painful to build and made them fun. It took user interactions that were previously reserved for large teams of experienced developers and put that power in the hands of small, nimble teams and newer developers.

Ten years after its inception, Rails now represents the 80% use case for web applications, with the 20% use case invovling highly interactive, app-like experiences.

The new table stakes

A decade ago, my water delivery company could get away with a static site with their phone number on it if you wanted to sign up or schedule a delivery.

Now, I'd take my business elsewhere if I couldn't manage my delivery schedule online. The stakes simply went up. My expectations as a user went up.

I'll contend that we're on the cusp of a new "80% expectation": the majority of people using our software on the web are now becoming accustomed to richer, more app-like experiences. The stakes are being raised.

Ember's bet on the future

Many of the benefits above are just about making it easier to do the same things we did before. But that's not Ember's bet. Ember's gamble is that when armed with this set of tools, you'll attempt increasingly ambitious projects.

Browser as an App Platform

I can't speak for the team that created Ember, but I can say from my vantage point that it looks like Ember is betting that the browser will act as an application runtime, not unlike the platforms for iOS or Android, and that the web is becoming the world's largest and most open app marketplace.

Interchangeable, Reusable Components

Web Components are an important upcoming change to the way web developers work, and Ember's take on them is impressive. When you start using Ember Components, it feels like the jQuery UI of the future: You get reusable drop-in functionality that's totally customizable, and backed by Ember's model system.

Ember Components are worth the price of admission alone, and we do drop them in as standalone pieces of interactivity into some of our server-side apps when it isn't feasible to create a full Ember application.

Ember in the real world

Its rather audacious goals are why Ember is improving so quickly, and the things people are building with it are sometimes mind-boggling.

Here are just a few of the cool things I've seen come out of the Ember community lately:

NBC News: NBC's new interface is more interactive exhibit than static news website
Huboard: A lovely kanban board built on top of GitHub issues
Discourse: No less audacious a goal than to replace every PHP forum on the internet
Ember Beats: Super fun drum machine webapp
Bustle: A women-focused news and opinion site
Vine: Twitter's 6-second video site

The limitations of the Web have historically been in its underlying technologies, but with advances like HTML5, CSS3, and tools like Ember, our biggest limitation on the web is quickly becoming our own imaginations.

Challenges

Challenge 1: The learning curve

Learning Ember was a challenge for me. I started pretty early and the framework was famous at the time for frequent, major API changes and sparse documentation.

Those have both largely been addressed with great documentation and things like Code School, and I see newcomers picking things up about 5 times faster than I did. (Here's hoping that speaks more about the state of Ember than it does my learning abilities.)

That said, it is a big library and can still feel still quite challenging to learn, and may actually be more difficult for programmers who are skilled at server-side MVC and must "un-learn" the way their framework handled working with the front end.

But once you grasp a few core concepts, you'll start to "get" Ember, and we'll outline those in another post.

Challenge 2: JS solution fragmentation

Tools for languages like Ruby are now mature, stable platforms, while JavaScript, by comparison, can feel like the wild west.

If you want RSpec as your test framework in Rails, it's generally one or two lines of code to make the change. Choosing and setting up your test setup in JavaScript is still confusing and often fraught with issues.

The good news is that people are working to solve this problem as well. Ember App Kit is one of the most ambitious projects I've seen in this space. It's young but it seems to aim to make choosing and setting up all the ancillary stuff as simple as it is in server-side frameworks like Rails.

So why Ember, specifically?

Ember, like any tool or framework, doesn't offer the perfect solution to every problem. But recently, whenever I've built an application without it, I've wished I had it to manage the complexity that lurks behind nearly every front-end feature request.

Between Ember's bulletproof object model and thoroughly-thought-out routing system, stuff that would have looked like a sheer cliff face before now looks like a playground to me.

It's the epitome of a disruptive technology: Ember allows me to do things that I wouldn't have even thought to attempt before, because the degree of difficulty was so high my brain just closed the door on the option. (Paul Graham calls this schlep blindness.) It begs to be given a harder class of user experience problem, which is fun, rewarding, and in line with our purpose at The Frontside to push UX on the web forward.

Lastly, I tend to vote with community. I think Ruby and Rails has emerged as one of the best communities in any category, not just programming, and I feel similarly about the fast-growing, yet tight-knit Ember.js community.

What if I'm wrong?

What if my crystal ball is broken, and users stay content with web applications as they are?

I can't deny that I really do believe we're looking at the future of web applications, but none of this future-talk matters too much. The simple point is this: Ember makes it fun to build things that are awesome.

And if you're going to do something, why not do something awesome?

Ember may not be for every problem or every developer, but if you'll give it a shot on your next project with high front-end ambitions, I promise you'll start feeling less intimidated and more like this:

Ember on Rails: #REALTALK

Brandon Hays — Sat, 21 Sep 2013 12:00:00 GMT

If you work in Rails and have ever wondered about Ember.js, you should know that Ember and Rails go together like Nutella and pretzels. (Which is to say, quite well indeed.)

Get an inside look of the experience of going from having never tried Ember to shipping a production application in it. What makes Ember a good match for certain types of applications?

This talk is intended to help relative newcomers jump in and start seeing the kind of crazy-ambitious applications you can build when you've got Ember.js in your toolset.

Note: The audio is rough, but gets a little better as the talk progresses. Sorry about that.

The Ruby Racer Rides Again

Charles Lowell — Tue, 04 Dec 2012 12:00:00 GMT

It began during RailsConf.

My fingers were itching to code, so between sessions I started tinkering with some of the more fanciful enhancements to The Ruby Racer I'd been contemplating as well as wrestling with a number of long-standing bugs. But what started out as a small refactoring slowly but relentlessly gained momentum until things were completely out of control.

By all accounts my hands were steady and the work proceeded in an orderly fashion. Instead, it was I who was out of control. Personally and psychologically I was helpless to stop myself as line after line, class after class unravelled before me --each one more rapidly than before. Soon a critical mass was achieved and boom! Nuclear. A from-scratch rewrite was underway.

Yes it was a rewrite, but by the time I was aware of it, I was too deeply invested to turn back.

It's a tichy thing, is a rewrite. Instead of the careful, iterative approach that serves as law in our profession, you drop everything and sprint towards the cliff in the hope that you can lunge across the chasm in in one leap. Once you're in the air there's no going back, and the possibility is as thrilling as it is real that, like Icarus, you'll hurtle into the abyss... another unsung casualty of hubris.

The prize of course is a better library that is more stable, has more features and is easier to build and install, so if a rewrite it had to be, I did the only thing I could do yield myself to it utterly.

I'm happy to say that it's mostly there there and that, despite some outstanding flaws, I'll be releasing the next version shortly. But more on that later. First, what's new?

Stability

For certain applications therubyracer has been plagued by crashes and memory leaks that have limited its usefulness to a much smaller domain than necessary: specifically, low-traffic applications that only instantiate a single JavaScript context. These problems stemmed from within the library but also from within Ruby itself.

The Ruby Racer relies heavily on weak references in order to maintain referential integrity between Ruby object and the JavaScript objects they represent (and vice-versa). Sadly, the WeakRef implementation on 1.9 is completely broken. This is unfortunate because functioning weak references are sometimes a necessary component for building correct programs on platforms with garbage collection. Luckily there are workarounds which have been deployed while we wait for 2.0.

But that's not the whole story. It would be a shameful misrepresentatio to lay the bulk of the blame on Ruby's door. Distributed garbage collection is hard, and The Ruby Racer engaged in all manners of evil hackery and dark GC foo in order to prevent cycles of garbage from spanning the Ruby and JavaScript VMs. Sadly, this did not work in the general case, and furthermore, was not portable to other Ruby implementations (or even Ruby versions).

I've come to realize that to solve these memory problems in the general case would require each host Ruby platform, as well as V8, to expose a uniform interface to its garbage collector so that The Ruby Racer can traverse its object graph in order to detect cycles of garbage. In other words "not bloody likely."

As a fallback, I've decided to throw up my hands and eschew GC histrionics in favor of more sane memory management in the C extension combined with an explicit teardown mechanism for cases where cycles of garbage do occur. e.g.

context = V8::Context.new
context['cycle'] = context #oh no, a vicious cycle!

context.dispose() # Gordion knot is cleaved, sir!

In exchange, we get a kinder, gentler racer that works on MRI as well as Rubinius, and which you'll be able to rely on in your production processes.

Resource Constraints

Speaking of production, the upcoming release contains a much more comprehensive coverage of underlying V8 functionality than every before.

Among other things, you can query the V8 vm for heap usage by young generation, old generation as well as absolute heap size including executable memory. Not only that, but you can also place hard caps on each of these numbers. That way you can contain memory usage and keep your processes under control; even in a resource constrained environment like Heroku.

A (hopefully) kinder build

One of the sorest pain-points with The Ruby Racer was the way it built. Prior to 0.11, it had a hard dependency on the libv8 rubygem in order to provide the actual V8 library. In retrospect, this was a bad idea.

If you were unable to build the v8 contained in the libv8 gem, then you were completely hosed. You could not use therubyracer at all. Or, if you wanted to use a custom v8 patched for security or with tweaked startup data, then you were out of luck. Finally, it raised an unnecessary barrier to distributing therubyracer as a binary gem (which I would like to do more of). If you have a binary version, then why do you need to have libv8 at all?

For these reasons, the dependency on libv8 has been converted to a soft one. If a compatible version is detected in your rubygems, then it will compile and link against it. However, if for whatever reason, you do not want to use the libv8 gem to build, then that is now your prerogative. The Ruby Racer will search your system for v8 just like any other library.

The only difference is that if you want to use the libv8 gem, then you will have to explicitly require a compatible version in your Gemfile.

More bugs, but not on Rubinius

There are still some outstanding issues on MRI that emerge under a heavy stress of rapid object allocation and deallocation. As far as I can tell they are related to to the weakref implementation, wherein MRI gets confused and starts handing back references to completely unrelated objects. I say "as far as I can tell" because the subtleties involved lie beyond the frontier of my understanding of MRI internals. Given my current workload, it is unlikely I will have time to make the necessary investment in understanding required to completely quash these problems.

Despite this, I'm releasing the new version anyhow in order to support Rubinius more fully and because I think that even on MRI it will solve more problems than it introduces.

For starters, 0.11 is very stable on Rubinius and that platform is long overdue for a version of therubyracer that doesn't suck. The Rubinius core team, and @dbussink in particular, have been very patient and helpful in helping me resolve all the lingering crashes both big and small... even going to the trouble of adding checks and diagnostics to the garbage collector just for my sake. They were even willing and often able to diagnose problems I was having on MRI in order to make the library for everybody. If anything, I feel that they deserve a functioning JavaScript environment and it wouldn't be fair for me to withhold it from them.

On the MRI front, I still think that closing some of the loopholes in the memory management, the enhanced extensibility, and better build make it worth releasing anyway. I've been recommending people to upgrade to the beta for almost 3 months and it is almost always the right decision. Given that I have neither the time nor the inclination to support what I now consider to be the legacy release, I think it's high time to either shit or get off the proverbial pot.

If you, or anybody you know has a deep knowledge of MRI internals and would be willing to help me address these issues as well as others that may arise, I would love to work with you to knock 'em down.

So this is why I'm releasing version 0.11.0 of The Ruby Racer today. Of course, I may come to quickly regret it, but that's what gem yank is for right?

Implementing a Jenkins Extension Point with the Native Java API inside a Ruby Plugin

Charles Lowell — Mon, 16 Jan 2012 12:00:00 GMT

In which I elaborate why the idomatic Ruby API is sometimes not enough, and describe a method to harness the full power of the underlying Jenkins API while still happily coding your extension in Ruby

The Ruby API

One of the primary goals of the effort to bring Ruby to Jenkins is to enable developers to extend Jenkins in a way that looks and feels like a normal Ruby environment. This means using rake, bundler ERB, and plain old Ruby objects to roll your plugin.

For example, the native BuildStep class is made available through the Ruby BuildStep. They are similar to be sure, but the Ruby object is much less complicated, and actually bears no relation to its Java equivalent inside the Java hiearchy of inheritance.

Exposing this simplicity is a worthy goal, but to do so requires careful consideration of each Jenkins Java API and how to provide its functionality in the Ruby way --no mean feat given the breadth of the Jenkins.

Sadly, it means that these Ruby-friendly APIs must necessarily lag behind their Java counterparts.

This can be a serious source of frustration for those looking to dive into Jenkins Ruby development right now. Initial excitement is quickly dulled when you find out that the extension point that you wanted to implement is unavailable from Ruby land. You might get discouraged and feel like you might as well be coding your plugin with Java.

Well don't lose heart! I'd like to share with you a super easy way to write your extension points in Ruby even when there isn't a friendly wrapper. We'll actually implement a very handy extension point called RunListener within a Ruby plugin even though it is not part of the official Ruby API. We'll do it by scripting the Java class directly with JRuby's Java integration feature, and then register it directly with the plugin runtime.

The Extension Point

We'll be working with the Jenkins RunListener interface. This is a wonderful extension point that allows you to receive callbacks at each point during the actual running of a build. There's currently no nice ruby API for it, but we won't let that stop us.

First, let's create a new plugin called my-listener. We'll do this with the jpi new command.

legolas:Jenkins cowboyd$ jpi new my-listener
    create  my-listener/Gemfile
    create  my-listener/my-listener.pluginspec

Fun fact: 'jpi' is an acronym for (J)enkins (P)lug-(I)n. You can install the tool with rubygems: gem install jpi

Next, we'll cd into our new plugin and create our listener class inside the models/ directory. Jenkins will automatically evaluate everything in this directory on plugin initialization.

legolas:Jenkins cowboyd$ cd my-listener/
legolas:my-listener cowboyd$ mkdir models
legolas:my-listener cowboyd$ touch models/my_listener.rb

Our ultimate goal here is to implement a RunListener, so let's go ahead and start our class definition inside that file.

class MyListener < Java.hudson.model.listeners.RunListener
  def initialize()
    super(Java.hudson.model.Run.java_class)
  end
end

There's a couple key takeaways here. First, notice that we use JRuby integration to extend the class hudson.model.listeners.RunListener directly. Second, and this is a gotcha anytime you extend a Java class: you must invoke one of the Java super constructors if it does not have a default constructor. I can't tell you how many times I've been bitten by this.

In our case, the RunListener class filters which jobs it will provide callbacks for by class. By providing a more specific class to the constructor, you limit the scope of jobs you'll receive to subclasses of that class. For our purposes, we cast a pretty wide net by selecting all builds via the AbstractBuild Java class.

Pro Tip: when you're implementing a native Java API, it really helps to have the javadoc open in one window so that you can view the documentation and crib from the source

Now that we've got our class defined, let's implement some methods! We'll add callbacks for when a build is started and when it's completed.

class MyListener < Java.hudson.model.listeners.RunListener
  def initialize()
    super(Java.hudson.model.AbstractBuild.java_class)
  end
  def onStarted(run, listener)
    listener.getLogger().println("onStarted(#{run})")
  end
  def onCompleted(run, listener)
    listener.getLogger().println("onCompleted(#{run})")
  end
end
Jenkins.plugin.register_extension MyListener.new

And finally, on the last line, we actually inform Jenkins about the existence of our new Listener with the call to Jenkins.plugin.register_extension

And that's about it. We can start up our test server with our jpi tool to see our listener in action.

legolas:my-listener cowboyd$ jpi server
Listening for transport dt_socket at address: 8000
Running from: /Users/cowboyd/.rvm/gems/jruby-1.6.5/gems/jenkins-war-1.446/lib/jenkins/jenkins.war
...
Jan 16, 2012 12:46:15 AM ruby.RubyRuntimePlugin start
INFO: Injecting JRuby into XStream
Loading /Users/cowboyd/Projects/Jenkins/my-listener/models/my_listener.rb
INFO: Prepared all plugins
...
INFO: Jenkins is fully up and running

To view the output, create a freestyle build called HelloWorld that doesn't have any build steps at all, build it and view the console output. You should see something like this:

Started by user anonymous
onStarted(#<Java::HudsonModel::FreeStyleBuild:0x2870068a>)
onCompleted(#<Java::HudsonModel::FreeStyleBuild:0x2870068a>)
Finished: SUCCESS

The Sweet Reality

Even though we were dealing more directly with Java, we were still able to use all of the simplicity that comes with developing plugins in Ruby. Furthermore, even though our extension point was just scripted Java, it doesn't mean that inside its methods it can't call as much pure Ruby code as its heart desires.

I hope that if you're considering writing your next (or your first!) Jenkins plugin in Ruby, you'll feel confident that you can always fall back to the native APIs at any point.

Ruby for Jenkins Goes Pre-Alpha

Charles Lowell — Tue, 13 Sep 2011 12:00:00 GMT

To quote Dave Bowman, "something wonderful happened" last week during the weekly Jenkins-Ruby hack session.

We were able to boot a plugin written in pure Ruby into a Jenkins server just by executing a single command from the command line (rake server). Furthermore, this command did not involve a compilation step, and the plugin that it booted did not bare a trace of evidence that it was to be loaded into a server written in Java.

That's right. Using nothing but good 'ol Bundler and Rake, we hoisted a pure Ruby plugin into the Jenkins runtime. And I can tell you that after hacking on this for almost a year: it felt... so.... good.... man.

It's not time to declare victory yet, but at least the first two of our major goals have been realized, and that is definitely a cause to celebrate.

Practically, this means that rubyists can check out the source and get hacking without needing any knowledge of Java or any Java toolchain. That's why I think it's safe to officially declare the project to be in a pre-alpha state.

It isn't yet for the faint of heart, and you're sure to find many things confusing and difficult. But at least those who know only Ruby can contribute bug reports, identify pain points, and contribute to the creative process.

It's 8:20am, but it still feels like Miller time.

The Ruby Racer isn't threadsafe... yet.

Charles Lowell — Mon, 13 Jun 2011 12:00:00 GMT

~~In which I explain problems unique to running The Ruby Racer in a multithreaded environment, which users are affected by these problems, and what is to be done for them.~~

UPDATE: This issue has been resolved with version 0.9.1. You should be able to stop reading here, update your Gemfile, and have a great day. That said, please don't let me discourage you from continuing down the page. It's riveting.

There have been a number of crashes reported in several different places for which The Ruby Racer is the culprit. For the most part, these are people running an Rails 3.1 application in a multithreaded webserver such as WEBrick, but as we'll see, it could be anybody that is using The Ruby Racer in a mulithreaded environment. In order to speak to these and future issues in one place, as well as prevent the spread of any misinformation or impressions, I wanted to very quickly explain what exactly is causing these crashes, who is affected by them and where, and finally how we're going to make them go away.

Background

When an instance of V8 starts running, it associates all of its global state with the currently running native OS thread. By global state, I mean things like heap storage, stacks, callbacks... everything that it needs to make JavaScript happen. Every time you eval() a script or you allocate an object with new, it retrieves that state from the executing thread to perform that operation.

In a single threaded application, that's the end of the story. This is because every V8 operation happens from the same thread as the rest of your code, so the V8 state is always right there ready at all times. If you're running your app in a rack server like Passenger or Unicorn that uses a multi-process model to achieve concurrency, you'll never see this crash because you don't use threads. Also, if you're running on MRI 1.8.7 which uses green threads, you will also not be affected. While you may have more than one Ruby thread in your application, you still only have one native OS thread which is what V8 will use.

The problem comes in when you are using Ruby 1.9 in a multi-threaded environment and you access V8 from more than one thread. This is because 1.9 uses native OS threads under the covers for its own threading API. When The Ruby Racer starts up V8, it uses whatever thread happens to be running as the home for its runtime data. Then, when you try and do some JavaScript operation from another thread, V8 can't find its state, freaks out and crashes your entire process.

Luckily, V8 does give us some help here. It provides a locker API as a way to deal with threading issues. It acts as both mutex on a V8 instance while at the same time providing a facility to access it from different threads. We can wrap calls that touch V8 with a Locker in order to prevent these crashes. e.g.

V8::C::Locker() do
  #... code that calls V8
end

That sounds simple enough: just never access V8 without a lock and you're good. In fact, ExecJS does this whenever it invokes its therubyracer runtime, and it works... almost.

I would have gotten away with it too if it hadn't been for you pesky GCs

If ExecJS locks V8, then why am I still crashing, and at seemingly random intervals?

The problem comes when we need to run garbage collection. The awesome thing about The Ruby Racer is that it lets you hold on to V8 objects from Ruby. It does this by using a proxy --a ruby object that implements all its behavior by delegating to an underlying JavaScript object. As long as Ruby holds a reference to that proxy, the proxy will in turn hold a reference to its underlying V8 object. However, when Ruby no longer holds a reference to the proxy and the proxy is garbage collected, we need to tell V8 that we aren't holding a reference to the underlying object anymore. And, as you may have guessed, in order to remove references to V8 objects, you need to hold what? Yep, the V8 lock. To operate properly, GC also needs to lock V8. Sadly, it does not, which explains why crashes continue to happen at random places.

You might be asking yourself, "why not just lock V8 in your GC routines and have done with it?" The answer is that it would be a terribly dangerous and irresponsible thing to do.

Suppose that thread A has V8 locked and is doing some fancy JavaScript stuff. At some point, we do a context switch over to thread B which is not doing anything even remotely JavaScriptful like loading ActiveRecords from the database or something. Thread B is chugging threw lots of memory, and all of a sudden triggers GC. There are some V8 objects that need cleanup due to our JavaScript activities over in thread A, so it tries to lock V8, but uh-oh, thread A already holds the V8 lock, so GC may have to wait forever until it becomes available. Deadlock!

The answer is that instead of immediately releasing v8 objects inside the GC thread, we need to equeue them somewhere where they can be released. This probably means starting a thread per V8 instance to consume this reference queue and release them in the context of a V8 lock that isn't contending for any other resources. While the answer is not trivial, I don't expect it to be that complex.

What to do? What to do?

In the mean time, there is a workaround for locking V8 with each request, which will ensure that GC running in that request will have already acquired the V8 lock. Bear in mind that this is not a silver bullet, and you still could encounter crashes and deadlocks, but they should be very few and far between. If, as is the case with most people, you are just using a multithreaded server in your development environment but deploy to multi-process model, then this should be sufficient. If on the other hand you run multi-threaded in production, then you should definitely not be locking V8 in your middleware since this will effectively synchronize every request. Not a good idea.

No matter which category you fall into, I expect that we'll have a beta fix by the end of this week, and if all goes well, a patch release after another. This may seem like a long time for such a small thing as coordinating locking with GC, but my experience tells me that threading is not a thing best done by humans, and so there will inevitable stumbles and cursing.

Also, I'm keen to make sure that any solution is both invisible to you the developer, as well as compatible with other Ruby versions and implementations. This deserves some thought.

What? you're still here?

Alright, time to stop talking and get started with the repairs. I just wanted to take a moment to let you know what the problem was, why it was happening, how're you're affected, what you can do now, and when you can expect a more permanent fix.

What it takes to bring Ruby to Jenkins

Charles Lowell — Thu, 12 May 2011 12:00:00 GMT

"Jenkins Ruby Plugins" are at an important, yet fragile stage of their life. Over the past several months, we have made tremendous progress towards making extending Jenkins with nothing but Ruby a blissful reality. However, at this moment, they exist only as a collection of hacks, or, to frame it more kindly, "proofs of concept" all buried quite deeply in a terribly named repo on github.

To name a few of them, we are now able to

automatically discover Ruby plugins and register them
work with Jenkins internals like BuildWrapper, RootActions, etc.. via a native Ruby-like API
persist mixed hierarchies of Java and Ruby objects transparently as Jenkins configuration
use ERB to template views, while still having access to the standard UI helpers
extend the Jenkins REST API from Ruby

This is by no means an exhaustive list, but I want to stress just how far we've come before I talk about where it is that we need to go.

While the work done so far represents some phenomenal accomplishments, in the end, they are only technical accomplishments which do not together create a coherent developer experience. So now, for this endeavor to succeed, we must focus our attention on making that entire experience, from project start to release after release, as frictionless as possible.

To that end, I propose the following high level goal for the Jenkins Ruby Plugins project:

To provide the facility to create, develop and release extensions for Jenkins with nothing but knowledge of the language, tools and best practices of the Ruby community.

This is, I think, the most important ideal to aspire to when claiming to provide a Ruby development experience. Unless it feels like the real McCoy, then you aren't really developing Ruby at all, you're just scripting Java with JRuby. You won't fool anybody, and there will be far less enthusiasm and adoption.

That said, several things follow logically from this thesis, the first of which is:

Absolutely, Positively, No Maven!

This is not an ideological stance on the value of Maven as a ~~wretched pile of toad sick~~ software development tool. Rather, it is an explicit acknowledgement that it is not a Ruby tool, it doesn't fit with the community's style, and if you're using it you might not be doing Java, but you certainly aren't doing Ruby. Instead, we should turn to the Ruby tools that occupy this same space: Rake for scripting the build, and Bundler for dependency management.

Ruby-Like Project Structure

If we aren't going to use Maven, then why should we be constrained by the project directory structure that it imposes. src/main/resources/what? Nope. Just one glance at Jenkins Ruby Plugin's layout in an editor or file system browser should put a Rubyist's mind right at ease. You see that? There's your Gemfile just where you left him. And oh, your pals lib/ and spec/ are there too. Let's get right to work!

Managing The Project Life-Cycle

From inception to release, you should be able to manage all aspects of your Jenkins plugin with Ruby. There must be single commands (or rake tasks) to

conjure a new plugin into existence
generate model/view boilerplate
run a server with the plugin loaded
package, push, tag and release a plugin to update-center

Testing

The importance of testing in the Ruby community cannot be overstated. The Ruby Tools need to provide a clear path to testing the plugin at both the unit and functional levels.

Documentation

The "Ruby-like" API that plugin development provides is similar, but not equivalent to its native Java equivalent. It is critical that this API be well documented, so that developers can understand it, and do not find themselves constantly referring to the JavaDoc, which they may very well not understand (it being Java), and might be misleading given the natural asymmetries of the two APIs.

Heed the Call!

There are plenty of people who would love to see this project get done, but don't know where to start. I know this because I was one of them. The upshot is that now, as I've just described, there is plenty of work to be done that doesn't require any knowledge of Java and very little knowledge of Jenkins internals. That means that you can contribute in a meaningful way!

Next week, I'll be publishing some specifications in the form of Cucumber scenarios for some of the features up above that we can discuss and then hopefully start implementing.

But the most important thing that you can do is to join in on the conversation!

irc, and email are just a few of the ways to reach out!

Update. I have posted the first scenario: creating a plugin. The first bounty is officially posted.

Wrapping each Hudson distribution in its own RubyGem

Charles Lowell — Thu, 06 Jan 2011 12:00:00 GMT

Update: As of version 1.396, this is applicable to the Jenkins project. 1.395 will be the last version of the hudson-war gem

As you may or may not know, we here at The FrontSide have been working since late last year to create a proper ruby environment for extending hudson. This is no mean feat because it implies not only the ability to use nothing but ruby code to implement hudson plugins, but just as importantly, the ability to use nothing but ruby tools to develop them. That means no javac, no jar files, no war files, and for the love of all that is holy: absolutely, positively, no maven. In short, the only folks you'll need to know to get in the door with Hudson are your pals Ruby, Rake and Bundler.

Of course, all those jars and wars and perhaps some class generation will be there behind the scenes. After all, we'll still need to compile java stubs that depend on core Hudson classes. We'll still need to package your plugin along with its ruby code and dependencies in a jar file, and we'll still need to take care of all the blah, blah, MANIFEST, blah, META-INF, WEB-INF, blah, blah, blah...

The War

If we're going to do the compilation against Hudson core from within ruby and Rake instead of maven, then we're going to need a reliable way to get at the core hudson classes so that we can use them in our compilation classpath. Not only that, but as part of the development process, you're going to want to test your plugins against a fully functioning hudson server. Luckily, both of these are contained in the full hudson distribution that you download as a single war file. Wouldn't it be nice if you had a simple way to get at a specific version of one of those wars and then get at all those java goodies contained therein?

The Gem

To my thinking, this sounds like a job for: RubyGems! That's why we're introducing the new hudson-war gem, which does nothing except wrap a single hudson distribution so that it can be managed coherently just like any other ruby component. Want the latest warfile?

gem install hudson-war

Or perhaps you need hudson 1.386? That's fine, hudson-war versions are the same as the hudson distributions they wrap.

gem install hudson-war --version 1.386

It's not just a war file either. In keeping with the best traditions of OOP data comes bundled with behavior. The gem comes with a ruby API and a CLI baked in to help us do all the nifty things we might want to do with a war file. For starters, you might want to know where it is.

legolas:~ cowboyd$ hudson.war location
/Users/cowboyd/.rvm/gems/ruby-1.8.7-p174/gems/hudson-war-1.386/lib/hudson/hudson.war

If you want to unpack it to a particular location then

legolas:~ cowboyd$ hudson.war unpack tmp
jar xvf /Users/cowboyd/.rvm/gems/ruby-1.8.7-p174/gems/hudson-war-1.386/lib/hudson/hudson.war -C tmp/

You can run a test server with it

legolas:~ cowboyd$ hudson.war server --daemon
Forking into background to run as a daemon.
Use --logfile to redirect output to a file
legolas:~ cowboyd$ hudson.war server --kill

Beyond the basics, let's say you want to do something interesting like compile some java code against the hudson core. You're gonna need a classpath, and you can get it with:

legolas:~ cowboyd$ hudson.war classpath
/Users/cowboyd/.hudson/wars/1.386/WEB-INF/lib/hudson-core-1.386.jar

You can even compile java code with it directly. Here we compile a trivial class that has a dependency on hudson.

legolas:~ cowboyd$ cat > ImportsHudson.java
import hudson.model.Hudson;
public class ImportsHudson {
  public static void main(String args[]) {
    System.out.printf("The Hudson Model object's class is %s\n", Hudson.class.getName());
  }
}
legolas:~ cowboyd$ hudson.war javac ImportsHudson.java
legolas:~ cowboyd$ java -cp .:`hudson.war classpath` ImportsHudson
The Hudson Model object's class is hudson.model.Hudson

It's like RVM for hudson. Wrangling java has never been so fun!

The War Gemmer

Befitting a continuous integration project, the Hudson team releases a new version at least every two weeks, and sometimes even sooner. It can be a bitch to keep up with, so I've set up a wargemmer hudson task to periodically poll the update center to see if there is a new release. If there is, it'll gem it right up and push it to rubygems. No fuss no muss.

Let the war begin!

We must come together to honor the command line

Charles Lowell — Sun, 14 Nov 2010 12:00:00 GMT

I reap no joy from writing command line interfaces in Ruby, and about 5 minutes before leaving RubyConf this year I realized that I am not alone.

It's not angst I feel so much as a general "meh": There's a fairly decent out-of-the-box way to do it in OptionParser, but the minute you want to introduce commands is when the muddling begins. Sure you can see your way through it, but it's never robust and you always get smacked in the ass by the edge cases that inevitably come along.

It ain't for lack of trying

My own abortive attempts not withstanding, I've tried in the last year at least 3 different libraries for parsing CLI options in my applications and gems. These are gli, thor and optitron. I have known and loved each of these libraries in its own way, but ultimately all of them left me feeling unsatisfied. I won't go into my specific dissatisfactions with each here, but suffice it to say that my malaise was shared. The general consensus of the cohort at the bar was that despite the recent innovations and improvements in the space, we'd really have just been better off with "OptionParser plus commands."

But we should do better

Better off for sure, but is that still what we want: to just get by?

No. we need a way to write command lines that feels like judo: deploying minimal force to effortlessly sling mountains of code.

I say we can have our cake and eat it too. I say our command lines can be easy. I say our command lines can be sexy. I say our command lines can be so awesome that they crap rainbows and have herds of unicorns come thundering out of their asses.

That's why I'm starting the optimal project. It may lead to nothing. It might even lead eventually to some code... that's not the point right now though. The point is to reach out to everyone who feels strongly about this and ask them to contribute ideas and requirements in order to build a consensus on the next generation of CLI builder that everybody can be proud of.

It certainly will not happen overnight, but for the betterment of all, it's high time to find the true successor to OptionParser: the one the community sees eventually integrated into the standard library.

Accessing Javascript Objects from Ruby

Charles Lowell — Mon, 25 Oct 2010 12:00:00 GMT

Awhile back, I wrote a post on how to access Ruby objects from inside your JavaScript enviroment when using The Ruby Racer. It showed just some of the many ways that you can access Ruby state and call Ruby code from within JavaScript. However, the story doesn't end there. The Ruby Racer is, after all, a two way bridge, and I thought it would be useful to document some of the ways in which you can access your JavaScript environment from within Ruby.

Let's start with the Context#eval method, which is used to execute JavaScript code from Ruby. Its the most intuitive way, and what gets used most often for examples.

Given a string, it compiles it as JavaScript source and then executes it inside the context. Nothing crazy there:

V8::Context.new do |cxt|
  cxt.eval('1 + 1') #=> 2
  cxt.eval('foo = {bar: "bar", baz: "baz"}')
  cxt.eval('foo.bar') #=> "bar"
  cxt.eval('foo.baz') #=> "baz"
  cxt.eval('new Object()') #=> [object Object]
end

Among other things, the context captures what functions and objects are defined in the global scope whenever anything gets eval()'d. So, for example, the Object constructor used in the last line is stored in the context.

As a tool for embedding however, eval() is the most blunt and (often) inefficient method available. For this reason The Ruby Racer, sports an extensive Ruby API for interacting with JavaScript objects which includes accessing properties, calling functions and methods, as well as creating new instances. In fact, I rarely use eval() at all to manipulate a JavaScript context except for loading source files into the interpreter.

The Ruby API consists of V8::Object and all of its subclasses:

V8::Context.new do |cxt|
  cxt.eval('new Object()').class #=> V8::Object
  cxt.eval('(function() {})').class #=> V8::Function
  cxt.eval('new Array()).class #=> V8::Array
end

Objects

The most fundamental operations in dealing with JavaScript objects are the getting and setting of values. Coincidentally, this is also a fundamental concept in Ruby, so it comes as no great surprise that we can re-use those constructs in Ruby that deal with property access to mirror those same operations on their JavaScript counterparts: the []()/[]=() and foo()/foo=() methods:

given the following context:

cxt = V8::Context.new
order = cxt.eval('order = {eggs: "over-easy"}')

we can read the eggs property of our order in several differnt ways. Via hash access (note that both string and symbol are acceptable as key values):

order['eggs'] #=> "over-easy"
order[:eggs] #=> "over-easy"

Values can be set with the hash-style access as well, and changes made from Ruby will be reflected accordingly on the JavaScript side

order['eggs'] = "sunny side up"
cxt.eval('order.eggs') #=> "sunny side up"

For property names that are also valid Ruby method names, you can access them just like you would with Ruby properties declared with attr_reader or attr_accessor. Again, changes made to the object in this way will be reflected in JavaScript:

  order.eggs #=> "sunny side up"
  order.eggs = "scrambled"
  cxt.eval('order.eggs') #=> "scrambled"
end

In the cases where the property name is not a valid Ruby method name, hash-style access is mandatory:

order['Extra $%#@! Mayonaise!'] = true

V8::Object also includes Enumerable and allows you to access all properties of a given object.

order.each do |key, value|
  puts "#{key} -> #{value}"
end

#outputs:
eggs -> scrambled
Extra $%#@! Mayonaise! -> true

As a convenience, V8::Context delegates the hash access functions to the JavaScript object which serves as its global scope.

order == cxt['order'] #=> true
order == cxt.scope['order'] #=> true

Arrays

Not much to see here, but before you move along: A V8::Array is like every other V8::Object except it has a length property, and it enumerates over the items stored at indices instead of the key, value pairs of its properties.

array = cxt.eval('["green", "red", "golden"]')
array.length #=> 3
array.map {|color| "#{color} slumbers"} #=> ['green slumbers', 'red slumbers', 'golden slumbers']

Functions

Consider the classic "Circle" example from every object oriented playbook you'll ever read. In JavaScript, the way to implement the circle "class" is with a constructor function.

circle = cxt.eval<<-JS
  function Circle(radius) {
    this.radius = radius
    this.area = function() {
      return this.radius * this.radius * Math.PI
    }
    this.circumference = function() {
      return 2 * Math.PI * this.radius
    }
  }
  new Circle(5)
JS

Now that we have the Circle constructor defined, and we have a reference to an instance of it in Ruby, we can call its methods just as though it were a normal Ruby object. Of course, only we know that under the covers the implementation is actually in JavaScript:

circle.class            #=> V8::Object
circle.radius           #=> 5
circle.area()           #=> 25Π
circle.circumference()  #=> 10Π

In JavaScript, methods are just object properties that happen to be functions. Therefore, you can get a reference to the actual function value just as you could from JavaScript simply by accessing the property by name. The resulting value is an instance of V8::Function.

area =                circle['area']
circumference =       circle['circumference']
area.class            #=> V8::Function
circumference.class   #=> V8::Function

Once you have a reference to a V8::Function, there are two ways to call the underlying JavaScript code (Actually there are three, but the third will be covered in the next section). These are the call() and methodcall() methods. To understand the difference between these two methods, it helps to understand how JavaScript functions themselves are invoked. In the event, there is the option to pass an object which will serve as the implicit invocant or this value. If no this is provided, the function will use the global scope in its place. Take for example the following JavaScript

var circle = new Circle(5)
var area = circle.area  //=> [object Function]
//no invocant, corresponds to ruby call()
area()  //=> NaN, there is no global 'radius'.
//call with invocant, corresponds ruby methodcall()
area.apply(circle)  //=> 25Π

The same code in Ruby:

area = circle['area']
area.class              #=> V8::Function
area.call()             #=> NaN
area.methodcall(circle) #=> 25Π

Because of this mechanism, JavaScript method invocation is much more flexible than Ruby in the sense that virtually any object can be used as the invocant of any function provided it has the requisite properties to satisfy the function's requirements:

area.methodcall(:radius => 5) #=> 25Π
other_circle = OpenStruct.new
other_circle.radius = 10
area.methodcall(other_circle) #=> 100Π

A very powerful construct indeed.

Constructors

But wait, there's more! Any JavaScript function can be either invoked normally, or, combined with the new keyword, as a constructor. It's what we used in the original eval() block to create the instance of Circle whose methods we were messing about with.

That was not quite necessary since we can invoke functions as constructors from Ruby too. This is done with the new method of V8::Function.

Circle = cxt['Circle']
circle2 = Circle.new(3)
circle2.radius        #=> 3
circle2.area          #=> 9Π
circle2.circumference #=> 6Π

Interestingly, when used as a constructor, a JavaScript function is almost completely indistinguishable from a Ruby class. This apparent duality between classes and functions is behind the decision to represent Ruby classes reflected into JavaScript as functions

The driving goal in all of these cases is the ability to author intuitive, tightly bound JavaScript, which necessarily doesn't involve evaluating strings to get what you want done. In closing, if there is something missing from this API you think would be useful, I would love to hear about it

RYE: Repeat Yourself Enough

Charles Lowell — Mon, 29 Jun 2009 12:00:00 GMT

Lately, I've been practicing the exact opposite of the DRY principle. Yup, you read that right: I repeat myself constantly, and in as many different contexts... including code, database schemas, test-plans and even documentation. The reason: So I won't have to repeat myself. It's not that I don't buy into the DRY principle, or that I'm somehow skeptical about the value its proper application yields. I just find it difficult to achieve in practice.

According to the Pragmatic Programmers who coined the term:

DRY says that every piece of system knowledge should have one authoritative, unambiguous representation. Every piece of knowledge in the development of something should have a single representation. A system's knowledge is far broader than just its code. It refers to database schemas, test plans, the build system, even documentation.

Essentially it is an empirical measure against which to gauge the proper level of abstraction in your system. If you have not made the right abstractions, then you will be found to be repeating yourself, whereas if you have made the right abstractions, then you won't. This is because abstractions directly encode the system knowledge that DRY talks about no less than 3 times in its definition.

The benefits of good abstractions are many and well known, and I'm not going to go into them here (there's always the internet for that), but what I've taken particular heed of lately is that abstraction is a double-edged sword that should not be wielded lightly. Get it right, and the previously intractable dissolves to simplicity in an instant... but get it wrong, and a snarl of complexity gridlocks your code base just as fast.

You see, I wasn't telling the whole truth when I said before that abstractions directly encode system knowledge. It would be more correct to say: Good abstractions directly encode system knowledge. Bad abstractions directly encode system ignorance.

The problem is that the DRY principle, while still very valid and useful, doesn't help us here because it assumes system knowledge as its input. But really valuable system knowledge isn't just lying around, it must be carefully constructed with significant amounts of research and analysis. That is to say, it must be learned. And what better way is there to learn something than by doing it over and over again until you know it inside and out? What better way is there than trial and error to really get at the physics of a phenomenon until you can use it to easily analyze it in all its forms? The fact is, you've got to repeat yourself over and over enough times until you can say with confidence: This I know to be true.

You can think of it like doing your math homework, or being like Daniel-San: waxing Mr Miyagi's car again and again until you know karate by instinct. In fact, I'm convinced that there's no other way to generate real knowledge.

Ironically, I repeat myself a lot these days because I hate repeating myself. I loathe it with an almost pathological passion, and sometimes (more often then I'd like to admit) it plays to my disadvantage. In my obsessive/compulsive rush to not repeat myself I'll introduce a bad abstraction based on flawed or incomplete system knowledge, and worse still, I won't know that I took the wrong fork in the road until it's way late in the game and it's hard to find the way back home.

That's why I'm playing it cool lately and making sure that I repeat myself enough to perfect the system knowledge required to keep myself DRY.

In Search of a Better Autogrowing Textfield

Charles Lowell — Tue, 16 Dec 2008 12:00:00 GMT

Autogrowing textfields are a nice piece of flash for any web application. They conserve precious screen real-estate, but grow to a height just big enough to accomodate the text the user actually typed without resorting to unsightly things scrollbars.

There are a number of solutions out there, but since I'm using jQuery on my current project, the two that I evaluated were Growfield and Autogrow.

While those two plugins often did what they were supposed to do, they also had tons of quirks and edge cases that prompted me give it a go myself. Specifically, I found that depending on the browser and operating system I would encounter:

eye-jarring jitter wherein the textfield would expand and contract my 1 or 2 pixels with every keystroke
resizing of the textfield to 0 pixels, when there was no text in the box
intermittent loss of focus on some browsers

But most importantly, it seemed that more often than not, the text did not actually grow correctly in any of the real-world environments that I tried in my production application.

As I discovered, the main reason for this was that the technique used was to fill a hidden DIV element with the text from the textfield, and then use the height of that DIV to determine how tall the textfield should be. That works great except for one nasty detail: most browsers use native widgets for text inputs, and those native widgets have their own word-wrap and layout code, so even things like font, line-height and letter spacing being equal, you can still get different word-wrap behavior from OS to OS.

In an attempt to side-step all of these issues, I thought "what if instead of using a DIV element to calculate the height, I could use a native textfield?" After all, if native textfields use a different word-wrap and layout algorithms, can't I just try and harness those algorithms, whatever they might be? At first this seems perfectly circular i.e.: I don't know what the ideal height of a textarea should be, so I'll use the height of a texarea to find out. Of course there's a twist, and the wonderful little property that makes it all possible is the scrollHeight.

lines, not heights.

Please Stop Using Global Variables in Ruby

Charles Lowell — Wed, 25 Jun 2008 12:00:00 GMT

I couldn't sleep this morning. I woke up around 4:30 AM thinking about how code is organized in ruby. I'm sure I will look back on my life with disbelief and a touch of shame at such behavior, but that is (hopefully) decades away.

The problem turning around in my head which eventually caused me to wake was my own inability to come to terms with a prevalent pattern in the ruby community: the global variable. Swirling in my semi-conscious mind was a motley mixture of bewilderment, self-doubt, scorn and a touch of righteous indignation.

On the one hand, there must be something I'm missing. One of the reasons I love ruby-space is that it tends to be filled with talented programmers who work the language to produce code that is powerful while at the same time being easy to understand and use. Surely, their collective conscious is a better guide than my own experience and intuition. I worry that is primarily my own fuddy-duddity that prevents me from accepting global variables as good practice.

To clarify, when I speak about global variables, I'm talking primarily about using global class objects as the repositories for methods and state. There are plenty examples out there of this:

# active record
Post.find(:all)

# amazon s3
S3Object.store('me.jpg', open('headshot.jpg'), 'photos')

# paypal-business gem
Paypal.capture(params)

In each of the preceding examples, there is implicit state stored on the class object. In active record, the database connection information, in S3Object, the AWS access key id and secret key, and finally, in Paypal, the paypal server url and business account name. Because the class object is global, the state on it is effectively global. And that is what sets my spidey sense to tingling.

Not only is this style subject to the all the arguments for why global variables are bad, but it just flat-out makes the API's themselves less useful. Here's an example from an actual project.

I had a system which needed to talk to two different SQS queues which were contained in two separate amazon web services accounts. Unfortunately, the standard SQS api uses class variables to store configuration information, and then a set of class methods to access the queue services, so I was forced to overwrite the global configuration parameters every time I wanted to access the queue. Luckily, the application was not multi threaded, or I would have been stuck rolling my own.

I don't see why:

SQS.access_key_id = 'ACCESS_ID'
SQS.secret_access_key = 'SECRET'
q = SQS.get_queue 'MyQueue'

is any better than:

@sqs = SQS.new :access_key_id => 'ACCESS_ID', :secret_access_key => 'SECRET'
q = @sqs.get_queue 'MyQueue'

(IMHO it's uglier) and in the second, hypothetical API, you can painlessly, and in a thread-safe manner talk to multiple queues on multiple accounts.

@one = SQS.new :access_key_id => 'AccountOne', :secret_access_key => "AccountOneSecret"
@two = SQS.new :access_key_id => 'AccountTwo', :secret_access_key => "AccountTwoSecret"

But there's a deeper principle at stake here than allowing multiple instances. It's about giving power to the programmer, and letting them control the API, and not the other way around.

It seems to me that when you're designing an API, you want the programmer(to the greatest extent possible) be able to create their own little world over which they have complete control. The should be able to spin up as many instances of your code and use it in ways that perhaps you hadn't thought of without them having to worry that one world will interfere with another world they've created.

If you've absolutely got to go global, it's very little effort to wrap a static/global interface around a single default instance, but not the other way around. Given the similarity in effort, it seems like embedability is the way to go.

Global variables surely have their place in some applications, and I'm not opposed to them on purely ideological grounds, but ruby already has a construct for global variables... it's called the '$' sigil.

In your environment.rb, or wherever you can just put:

#environment.rb
$sqs = SQS.new :access_key_id => 'AccountOne', :secret_access_key => "AccountOneSecret"

#somewhere else
q = $sqs.get_queue "MyQueue"

My suspicion is that Ruby on Rails shares a large portion of the blame for the evolution of this style what with ActiveRecord.establish_connection() and friends. As the highest profile ruby project it's only natural that people will copy its conventions --for better or for worse. Still, I wish people in the future would break from this particular precedent and if they go global, to do it properly. Give the poor class objects a break!

Patiently waiting for JavaFX

Charles Lowell — Thu, 05 Jun 2008 12:00:00 GMT

About a year ago, SUN announced a "new" platform for developing embeddable applications on the web, and they called this platform JavaFX. In reality, this new platform seems to me less of an innovation and more of a rehabilitation of the applet infrastructure which they allowed to languish over the last decade.

That's not a criticism, it's a compliment. I'm happy that they're going to give applets another go. Heck, I'm even glad they're calling it JavaFX too. "Applet" always sounded too effete anyway. In any case I've always like the applet model, and wished it were a more viable option for web development. Given the success of Flex, which also uses the applet model, other people must feel this way too.

But I digress.

As far as I can tell (I've only been able to get my hands on some demos and a little open source compiler) JavaFX involves upgrading some of the more boorish aspects of the java browser plugin. To name a few: faster download and initialization of application and JRE code, tighter integration and communication with the rest of the DOM, modern video and audio codecs.... All those are necessary upgrades to ensure the viability of the platform, but they can hardly be called innovative

That said, one new and curious aspect of JavaFX is that it will (for the most part) not be written in Java. Instead, SUN is promoting a completely new and aptly named scripting language: JavaFX script.

Of all the new features it was the specification of this language that impressed me most. I'm not sure how this little gem of a language managed to come out of SUN, but it's absolutely loaded with modern language features to make programming in it clear and concise.

I'm talking about

closures
pure functions
lazy evaluation
list comprehensions (that put ruby's to shame)
optional declarative syntax
rolled in query language for searching and modifying data structures
data binding supported at the language level
tons of stuff I'm sure I don't even know about....

Overall it seems like JavaFX script is like 4 separate languages existing in harmony in one neat package, and as I read more and more about it, I kept thinking to myself "This is not just some standard UI scripting language with a little sugar and a few extra hooks for writing GUIs. This is positively avante-garde.

I like the idea that SUN can challenge their developers to think in new ways, but I can also imagine a backlash, or more mildly put, an general aversion in the developer community, since this is not any old language research project, but the centerpiece in SUN's product in the battle over the browser VM.

So where did this decision come from to go with such a departure from the norm when it came to defining JavaFX script? I like it alot, but I worry that SUN is going to chicken out, and go with something a little less sexy but a little more digestible (at least in the short term) to its current developers as well as the developers of competing platforms like Adobe Flex or Microsoft Silverlight.

mod_rewrite: Remeber that the version counts

Charles Lowell — Mon, 11 Jun 2007 12:00:00 GMT

Pro Tip: Which regular expressions work depends on the version of Apache/mod_rewrite that you're using. I recently tested the following rewrite rule on Apache2

RewriteRule ^archives/(\d+).html http://www.thefrontside.net/map2new.php?$1 [R]

I wanted to match files like archives/000532.html.

But when it came time to deploy, it didn't work. Turns out the environment I was deploying to was using a different version of Apache. That's bad practice of course, but aside from that, the little gotcha was that the regular expression interpreter was different in different versions of Apache, and in this case, its behavior differed not only from the newer version but also from the behavior of most regexp engine's out there. Specifically, the "digit" literal \d is not understood by older versions (which interpret it as a literal "d").

Instead I had to use [0-9]

RewriteRule ^archives/([0-9]+).html http://www.thefrontside.net/map2new.php?$1 [R]

This is a very specific nugget, but hopefully it will save someone a headache down the road. Just remember, if your mod_rewrite regexp isn't working, check the specific version of the engine, and make sure your regexp is one that it will understand.

Trans-Global Pair Programming

Charles Lowell — Sat, 09 Jun 2007 12:00:00 GMT

The Frontside Software is a three person company with "offices" in Michigan, Finland, Massachusetts, and New Jersey. We're don't see each other every day, and we're rarely in the same room, but we still do a significant portion of our development work in pairs. Despite many other competing setups, we still do this with the not-so-new, not-so-exciting, yet extremely flexible and reliable VNC combined with a voip product like skype.

How it works

One of us (the driver) runs a vnc server which transmits everything that is rendered on his display to one or more vnc clients (passengers) being run by the other half of the pair. That way, the client can see everything that goes on while the driver is coding, including his code editor, his web browser, his terminal windows, etc... Meanwhile, you've got real-time audio so that you can talk about the work you're doing as you're doing it.

Is there something better?

Not yet. We've seen some new collaborative coding tools like SubEthaEdit and Gobby come down with some very slick features. Specifically, the updating and syncing of editor state between the two machines is very fast, and effectively coordinates multiple people editing the same document, with as few clashes as possible. The way in which they do this is impressive, but after having given it several abortive attempts as a real solution for remote pair programming, we went back to good old VNC. Here's why:

Lack of editing features: The collaborative editors of today are good at one thing: editing text collaboratively. The problem is that when you're pair-programming, you're not editing text, you're editing code, and code is only a simple sequence of text to a computer. There are lots of editors these days that leverage the semantically rich structure of the documents on which they operate like TextMate, Emacs, Eclipse.... everybody has a favorite, and because your collaborative editor is not your favorite, that means it sucks ;-)
Lack of environment: Of course, there are a scant few collaborative editing plug-ins for existing IDEs which would seem to address this problem, but adding on another layer, development is about more than just coding. It's about browsing documentation, running servers, invoking build scripts from the command line, and about a million other tiny tasks. In effect, your actual IDE is not just one application, it's your whole computer, and if the only thing being shared is a single app, then it cuts your pair out of a lot of important context. With VNC, everybody sees what's going on all the time. They can see not only the code, but also the running program.
Editing the same document at the same time isn't really helpful anyway: If you've done much pair-programming, then you realize that the real value doesn't come from having two sets of fingers on the keyboard at the same time. In fact quite the opposite: Both participants are following the thread of development, but one of them is freed entirely from the act of coding so that they can think about high-level architectural issues, or lookup api docs, or google for resources --all in parallel. Having both people pounding on the keyboard actually hinders this dynamic.
VNC is cross platform: This is a biggie. There are vnc clients and servers for Windows, OSX, and Linux (we code on all three), and they all interoperate with each other. That's a pretty hard feature to top, especially for single-platform apps like SubEthaEdit and Gobby which run on OSX and Linux respectively.

About the only drawback to pair programming with VNC is that there can be some fairly significant lag (between 1-4 seconds) depending on network load on either end, but even so, when it comes to actually developing collaboratively, VNC and Skype are our daily tools of choice.

VNC Clients/servers

OSX client: Chicken of the VNC
OSX server: Vine Server
Windows/Linux client/server: http://www.tightvnc.com/download.html

Taming the Rhino

Charles Lowell — Fri, 04 Aug 2006 12:00:00 GMT

I recently described how to use the one of the freely available shells as a great way to explore your javascript runtime. There are two implementations of the javascript interpreter sponsored by the Mozilla project, Spider Monkey an interpreter implemented in C, and Rhino, an interpreter implemented in Java. With respect to the javascript runtime itself, these two implementations are almost identical, and so what works in one, will generally work in the other. They diverge, however, when it comes to embedding objects and functions that are implemented in a language other than javascript. Naturally, Spidermonkey is better suited for embedding objects implemented in C, while Rhino excels at embedding objects implemented in Java.

What exactly is "embedding" an object implemented in Java? If you don't know what this means already, I could try to describe it to you, but why not show it live and in the flesh with the Rhino command line? That's the exploratory technique that I find so valuable.

cowboyd@subzero:~$ java -classpath js.jar org.mozilla.javascript.tools.shell.Main
Rhino 1.5 release 5 2004 03 25
js> var javaString = new java.lang.String("Hello World")
js> javaString.hashCode()
-862545276
js> javaString.startsWith("Hello")
true
js> javaString.startsWith("World")
false
js> var jsString = new String("Hello World")
js> jsString.startsWith("Hello")
js: "<stdin>", line 5: uncaught JavaScript exception: TypeError: startsWith is not a function. (<stdin>; line 5)
js>

As you can see, javaString is a reference to an actual java.lang.String object, and has access to all the methods of that class, many of which are not contained in the native javascript String.

Embedding goes two ways. Not only can you instantiate and use java objects from javascript, but you can also pass in javascript objects as parameters to java methods. You can even extend objects and implement interfaces in javascript! Once again, the command line shows this in action. In this example, we'll implement the java.lang.Runnable interface in javascript

cowboyd@subzero:~$ java -classpath js.jar org.mozilla.javascript.tools.shell.Main
Rhino 1.5 release 5 2004 03 25
js> var impl = new Object()
js> impl.run = function() {print("Yeah that's right, you better run!")}

function () {
    print("Yeah that's right, you better run!");
}

js> var runnable = new java.lang.Runnable(impl)
js> var thread = new java.lang.Thread(runnable)
js> thread.run()
Yeah that's right, you better run!
js>

This is all well and good, but that's quite a bit of code to get right the first time! Chances are, if you're just starting out with the Rhino command line, you're not going to have as much luck, especially if you're using it as a tool to learn javascript in the first place, and unfortunately, this is where Rhino comes up way short. Rather than have a forgiving CLI, Rhino punishes you for every syntactic and semantic error that you make by not collecting command history. Even worse, hitting the up and down arrows result in bizarre character literals output directly to the prompt. You can't even correct a mistake that you've made in the current line you're typing without backspacing all the way to the error, and then re-typing from that point on.

In this example, I'd like to arrow-left so that I can correct my misspelling of "java.lang"

cowboyd@subzero:~$ java -classpath js.jar org.mozilla.javascript.tools.shell.Main
Rhino 1.5 release 5 2004 03 25
js> var r = new java.lng.Runnable(^[[D^[[D^[[D^[[D

Ugh! Or what happens if the shell didn't take my last command because it was slightly bogus? I'd like to retrieve the command with the up-arrow, edit it a little bit in-place and then try again because after all, it was only slightly bogus. Watch me try and recover from this minor syntax error...

cowboyd@subzero:~$ java -classpath /usr/share/java/js.jar org.mozilla.javascript.tools.shell.Main
Rhino 1.5 release 5 2004 03 25
js> var f = function() {print("oops I forgot to close these parens"}
js: "<stdin>", line 29: missing ) after argument list
js: var f = function() {print("oops I forgot to close these parens"}
js: ...............................................................^
js: "<stdin>", line 29: missing } after function body
js: var f = function() {print("oops I forgot to close these parens"}
js: ...............................................................^
js: "<stdin>", line 29: Compilation produced 2 syntax errors.
js>
js> //I know. Up-arrow to the rescue!
js> ^[[A^[[A^[[A^[[A^[[A^[[A //drat, foiled again!

These problems are in particularly nasty contraposition to the technique of exploration via the shell which I advocate because the the cost for failure is so expensive. Indeed, what is so wonderful about most modern shells is that the cost for a syntax error is so small. For some reason, the implementors of the Rhino CLI decided to implement their shell with the typical functionality circa 1962.

It's not a problem for me though, thanks to one of my favorite unsung java libraries, JLine. JLine hits a super sweet spot in that it takes somewhere around 0 effort to add loads of standard functionality to your command line interfaces. It seems that no one in the java world bothers with a decent CLI; tragic in my opinion, but probably because it's considered well-understood, non-trivial and therefore tedious. With JLine, building that CLI comes at around 0 cost. What's really cool about JLine is that a program doesn't even need to be written with it. It can transparently intercept the console input for any java program and seamlessly splice on any and all functionality you'd expect from a hot shell: in-place editing, command history, you name it. In a word: perfect for a beast like Rhino.

cowboyd@subzero:~$ java -classpath js.jar:jline.jar  jline.ConsoleRunner org.mozilla.javascript.tools.shell.Main
Rhino 1.5 release 5 2004 03 25
js> prnt("oops let me try that again")
js: "<stdin>", line 1: uncaught JavaScript exception: ReferenceError: "prnt" is not defined. (<stdin>; line 1)
js> print("oops let me try that again")
oops let me try that again
js> //trust me, that was easy. Just like it should have been in the first place.
js>

JLine truly is a healing salve for your chafing CLI woes. Did I mention that it's cross-platform?

While the barrier to entry is extremely low, the path to upgrade is but a mild upward slope. If you do end up wanting nice extras such as TAB-completion or custom key bindings, it has a simple configuration mechanism, and Java API to make the pain of writing custom code as minimal as possible. But that's another bedtime story altogether.

I hope you enjoy JLine with Rhino, or with any other impolite java command lines you may use!

Live Stylesheets

Charles Lowell — Sat, 15 Jul 2006 12:00:00 GMT

Even after working with DTML for over a year, I'm still constantly astounded by how dynamic it actually is. The most important thing to keep in the back of your head as a DHTML programmer is that the same mechanisms used by the browser to build an HTML document at load time are available to your in-page scripts. So, really, the static loading of your pages when you access a url is just the HTML parser invoking the same methods on the same objects that are available to your javascript code.

Take this static HTML code:



This is my div

The same result can be achieved dynamically:

Nothing mind-blowing there; indeed, this technique is the cornerstone of DHTML. But it takes awhile to let it sink in to the point where using it is one of your first strategies to solving a problem. Specifically, you can use it not only to dynamically change layout-related HTML elements (span, div, ul, table, et al...), but also to create/modify behavioral elements(link, script, meta, etc...)

That said, I was recently trying to create a CSS stylesheet at runtime, and enable it so that its rules would be active in the page. At first, I was trying to use the W3C DOM CSS interface, the idea being to create a stylesheet object, add a bunch of rules to it corresponding to the rules that I wanted, and then put it into the stylesheets array. Unfortunately, doing this in a cross-platform way is borderline impossible. After pounding my head against that for awhile and getting nowhere, I figured, "why not just create the whole thing as a top-level html element, and let the browser just take it away from there?" The basic strategy is this: create a style DOM Element dynamically, add text content to it representing the actual stylesheet, and then pop it into the HTML DOM. With a few caveats, it works like a charm. The browser parses the text as CSS, and links in into the live cascade.

I mentioned caveats.. well, of course there are the obligatory cross-browser compatibility issues to be aware of:

In Internet Explorer, you can't just append a Text Node to a Style element, you have to use some trickery to get it to work.
KHTML & Safari do not honor style elements that are not contained in the HEAD element of the document.
While IE and Mozilla always create a HEAD element for you if one doesn't exist, Opera and KHTML do not.
While Opera will honor a dynamically created HEAD element, KHTML will not. (That's a bug in KHTML as far as I'm concerned as it violates the DHTML principle that I've been talking about.)

All in all, they're nothing to worry about and do not stand in the way of the fundamental technique. I've created a demo page showing this code in action. As a nice side effect, it works as quick way of playing around with CSS properties and how they effect the styling of elements.

Here is the source snippet implementing the technique I've described here:


   var style = document.createElement('style')
   style.setAttribute('type', 'text/css')

var cssText = $('cssText').value if (style.styleSheet) { //IE only style.styleSheet.cssText = cssText } else { //for some reason this fails in IE. var text = document.createTextNode(cssText) style.appendChild(text) }

Learning Javascript from the Command Line

Charles Lowell — Sat, 08 Jul 2006 12:00:00 GMT

In the Drunk and Retired Podcast, episode 59 I spoke about learning your way around javascript the language independently from the browser, and how you can use the command line tools that come with the various javascript engines to interactively explore the javascript runtime.

When most folks think about javascript, they think about scripts that they embed into their web pages, but the truth is that it is a general-purpose programming language that has absolutely nothing to do with HTML. In fact, the javascript runtime is so orthogonal to other web browser functionality, that mozilla offers the javascript interpreter that it uses in Firefox and friends as a completely separate download. It's available as both .deb or .rpm package, and just to show it: Here's the wonderful hello world program, as entered into the shell.

cowboyd@subzero:~$ js
js> alert('hello world')
1: ReferenceError: alert is not defined
js>

OK, so I boobie-trapped that example in an attempt to beat the point I've been making to death. It's an error because `alert()` isn't actually part of javascript. In the context with which we're familiar(DHTML), it's a function that's defined by the browser. Of course, it just so happens that every browser implements `alert()` to behave in almost exactly the same way, but the function itself has nothing to do with the javascript core. Implementing our own version of alert is simple enough though.

js> var alert = function(message) { print(message)}
js> alert('hello world')
hello world
js>

Personally, I love the command line because it let's you dig your fingers deep into the computer's brain and, by pushing its buttons directly, see what's going to work and what isn't. Every time I have a question about how the javascript interpreter is going to behave, I don't look up the spec, or write something into my programs that I'm not sure how it will work. Instead, I fire up my trusty interpreter to discover empirically how the system works. Need to know if a RegExp is going to match? Don't guess. Ask the interpreter.

js> "foo".match(/bar/)
null
js> "foo".match(/oo/)
oo
js> "foo".match(/oo$/)
oo
js>

Wonder what the built-in "constructor" property of an object is? The runtime can tell you. It's his business after all.

js> function A() {}
js> var a = new A()
js> var o = new Object()
js> a.constructor

function A() {
}

js> o.constructor

function Object() {
    [native code]
}

js> o.constructor == Object
true
js> a.constructor == A
true
js>

Sure, you could write an in-browser script to do all this and throw output at yourself in the form of alerts, but the beauty of the javascript command line is that you can collapse the whole edit-save-reload-alert scripting cycle into a single step; type in the next line and see what happens. It's that super-tight feedback which let's you learn that much faster.

Frontside Software RSS Feed

Announcing Effection 3.0 -- Structured Concurrency and Effects for JavaScript

🏷️ Excellent TypeScript Experience

✨ New Context API

❤️ Embrace the JavaScript

Async

Effection

🧠 Rebuilt with Delimited Continuations

Get Started With Effection 3.0 Today!

The await event horizon in JavaScript

Is Explicit Resource Management a solution?

Does AbortSignal help?

Structured Concurrency and the await event horizon

Structured Concurrency and JavaScript

Backstage: Unblock Innersource Development with Plugins and Code Owners.

Backstage: How to fix linting errors without slowing down development

Writing the Patch

Viewing and Updating the Summary

Using the Data

Conclusion

Deno is the best tool for maintaining NPM packages

Tag once. Publish twice.

Stage 1: The release tag

Stage 2: Build and Publish

2.1: Publish to deno.land

2.2: Publish to NPM

Conclusion

Seven ways to build a single glass pane with Backstage

Background

Option 1: Explore Backstage Plugin

Option 2: Entity Links

Option 3: Custom React Component

Option 4: Proxy Backstage Plugin

Custom Backend Plugin as a Gateway

Option 6: Custom Processor

Option 7: Custom Entity Provider

Conclusion

Dynamic Github Action Jobs

Dynamic Github Action Jobs

Ingestion tests in Backstage Part 2: Safely refactoring an LDAP integration

Upgrade with confidence

Focus on what matters

Deno is "blazing fast" for humans

What Is Spotify Backstage: a developer portal overview

What can Backstage do for you and your team?

Technical speaking, what can we find in Backstage?

Software Catalog

Software Templates

Search

TechDocs

Plugins

Unleash the power of Backstage

Tutorial: Using Github Webhooks with Backstage Entity Provider

Adding an Entity Provider

Configure Github Webhook

What's next?

Testing Backstage Catalog's ingestion

Controlling the Server

Assertions Made Simple

The Dividend

Up Next

Backstage tutorial: plugin integration tests with Cypress + Interactors

What you'll be testing

Set up Cypress and Interactors

Assert with Interactors

Act with Interactors

Extend existing Interactors

Write your own Interactor

Other Material UI Interactors

Conclusion

Announcing GraphGen: the search for the graph factory

Tutorial: How to use your company’s component library with Backstage

Create a Backstage App

Clean up your app

Install your component library

Change the theme provider

Use Backstage Catalog API to fetch data and display it in a table

Make catalog API available in your application

Use useApi hook to get the CatalogApi

Use useAsync hook to call the CatalogApi

Use `useApi` hook to get the CatalogApi

Use `useAsync` hook to call the CatalogApi

Use `useMemo` hook to convert result from CatalogApi into items for `DetailsList`

Render `DetailList` with items

Rebuild `devcontainer`