Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(nx-dev): conformance rule for blog post description #29835

Merged
merged 3 commits into from
Feb 4, 2025
Merged

feat(nx-dev): conformance rule for blog post description #29835

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
889bf70
feat(nx-dev): conformance rule for blog post description
juristr Jan 31, 2025
7a5c7ac
docs(nx-dev): add missing blog description fields
juristr Jan 31, 2025
da06d92
fix(nx-dev): update blog metadata to include the post descriptions
juristr Feb 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 16 additions & 16 deletions docs/blog/2021-06-15-distributing-ci-binning-and-distributed-task-execution.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@ slug: 'distributing-ci-binning-and-distributed-task-execution'
authors: ['Victor Savkin']
cover_image: '/blog/images/2021-06-15/jFVfKEglfQIM9QsP.png'
tags: [nx]
description: "In this post we looked at two ways to distribute your CI: binning and using Nx Cloud's distributed task execution."
description: "Learn how to scale your CI pipeline using two distribution strategies: binning for workload distribution and Nx Cloud's distributed task execution for optimal performance."
---

As your Nx workspaces grow, running CI on a single agent becomes unworkable. Nx’s code change analysis and computation caching allows you to do the minimum amount of computation needed to verify that the PR is good to merge, but it only helps with the average case CI time. No matter how smart Nx is, in the worst case you need to rebuild/retest everything. **That’s why any sizable workspace has to distribute CI across multiple agents.**
As your Nx workspaces grow, running CI on a single agent becomes unworkable. Nx's code change analysis and computation caching allows you to do the minimum amount of computation needed to verify that the PR is good to merge, but it only helps with the average case CI time. No matter how smart Nx is, in the worst case you need to rebuild/retest everything. **That's why any sizable workspace has to distribute CI across multiple agents.**

In this post we look at two ways to do that.

Expand All @@ -19,15 +19,15 @@ Binning is an approach to distribution where the planning job divides the work i

Nx has always provided affordances to do that, and many workspaces took advantage of it. Most of the setups look similar. This is an [example of implementing binning using Azure Pipelines](/ci/recipes/set-up/monorepo-ci-azure).

The planning job invokes _print-affected_. This command executes the same logic as _“affected:\*”_ but instead of running the tasks, it returns the tasks’ descriptions. The job invokes this command for each target such as build/test/lint/e2e. After that, each worker agent runs the tasks assigned to it.
The planning job invokes _print-affected_. This command executes the same logic as _"affected:\*"_ but instead of running the tasks, it returns the tasks' descriptions. The job invokes this command for each target such as build/test/lint/e2e. After that, each worker agent runs the tasks assigned to it.

Binning is very common. For instance, [the CircleCI support for running tests in parallel](https://circleci.com/docs/2.0/parallelism-faster-jobs/) uses binning.

We at Nrwl helped many large companies distribute CI using different variations of binning. It works reasonably well for simple cases, but not without issues.

## Issues with Binning

### Binning doesn’t partition the work in the optimal way.
### Binning doesn't partition the work in the optimal way.

First, you cannot partition tasks into bins without knowing how long every task takes. Most binning solutions collect timings (including the one in the Azure example above) which works imperfectly.

Expand All @@ -41,7 +41,7 @@ If you partition your tests into five bins, you have five agents with separate l

More importantly, you often need all the file outputs for a given target on the same machine to do post-processing. For instance, **you can run the tests on 5 agents, but you need all the coverage reports in the same place to combine them and send them to SonarQube.** Doing this is challenging.

### Binning doesn’t work for builds.
### Binning doesn't work for builds.

Any time you run a command in a monorepo, Nx creates a task graph, which it then executes.

Expand All @@ -51,17 +51,17 @@ This is common when you have libraries that depend on other libraries.

![](/blog/images/2021-06-15/lS7eewNQuZzQ72kU.avif)

In this example, the Child 1 and Child 2 libraries have to be built first. Parent 1 can start only when Child 1 has been built because it needs the Child 1’s dist folder. Parent 2 has to wait for both Child 1 and Child 2. And they can all be built on different agents, so their dist folders will have to be moved from agent to agent. You cannot implement it using binning. This problem also occurs for tests that require the libraries or applications to be built first.
In this example, the Child 1 and Child 2 libraries have to be built first. Parent 1 can start only when Child 1 has been built because it needs the Child 1's dist folder. Parent 2 has to wait for both Child 1 and Child 2. And they can all be built on different agents, so their dist folders will have to be moved from agent to agent. You cannot implement it using binning. This problem also occurs for tests that require the libraries or applications to be built first.

That’s why you often see tool authors talking about distributing tests and not builds. **Distributing tests is relatively straightforward. Distributing builds is hard.**
That's why you often see tool authors talking about distributing tests and not builds. **Distributing tests is relatively straightforward. Distributing builds is hard.**

### Binning complicates CI/CD Setup.

Maintaining a CI setup that uses binning is often an ongoing effort. Because you don’t have a proper coordinator, your CI has to be the coordinator, which complicates things.
Maintaining a CI setup that uses binning is often an ongoing effort. Because you don't have a proper coordinator, your CI has to be the coordinator, which complicates things.

## Approach 2: Nx Cloud 2.0 Distributed Task Execution (DTE)

We at Nrwl are in the business of helping companies use monorepos, so we have been dealing with these issues for many years. Nx Cloud 2.0’s support for Distributed Task Execution is our solution for this problem. It solves all the problems listed above and more.
We at Nrwl are in the business of helping companies use monorepos, so we have been dealing with these issues for many years. Nx Cloud 2.0's support for Distributed Task Execution is our solution for this problem. It solves all the problems listed above and more.

## How Does Distributed Task Execution Work?

Expand Down Expand Up @@ -121,23 +121,23 @@ As you can see there is not much that changed. We added the agent job, registere

![](/blog/images/2021-06-15/XISTgZIBj5ZZ3Sp7.avif)

It won’t run the build locally. Instead, it sends the Task Graph to Nx Cloud. Nx Cloud Agents pick up the tasks they can run and execute them.
It won't run the build locally. Instead, it sends the Task Graph to Nx Cloud. Nx Cloud Agents pick up the tasks they can run and execute them.

> The Nx Cloud agents here are CI jobs that run `npx nx-cloud start-agent` so they can can be defined in any CI env.

This happens transparently. If an agent builds `app1`, it fetches the outputs for lib if it doesn’t have it already.
This happens transparently. If an agent builds `app1`, it fetches the outputs for lib if it doesn't have it already.

As agents complete tasks, the main job where you invoked `nx affected --build` l starts receiving created files and terminal outputs.

After `nx affected --build` completes, the main job has the built artifacts and all the terminal outputs as if it ran it locally.

Let’s reexamine the issues above to see how we addressed them.
Let's reexamine the issues above to see how we addressed them.

### Nx Cloud partitions the work in the optimal way.

In theory every agent could pull one task at a time to partition things evenly, but it doesn’t work well in practice. The network overhead can add up for very small tasks, and it’s often faster to run several tasks in parallel because of the batching capabilities Nx has.
In theory every agent could pull one task at a time to partition things evenly, but it doesn't work well in practice. The network overhead can add up for very small tasks, and it's often faster to run several tasks in parallel because of the batching capabilities Nx has.

As you run commands in your repo, Nx Cloud collects the timings and uses those to partition the work into well-sized batches, such that if one agent is slow, the CI isn’t blocked. Agents also run tasks of different types (tests/lints), so the pool of agents is shared evenly.
As you run commands in your repo, Nx Cloud collects the timings and uses those to partition the work into well-sized batches, such that if one agent is slow, the CI isn't blocked. Agents also run tasks of different types (tests/lints), so the pool of agents is shared evenly.

### Nx Cloud does not split commands.

Expand All @@ -151,7 +151,7 @@ To stress one more time the main job contains all the terminal outputs and all t

**Nx Cloud is a proper coordinator and it can process any task graph**. An Nx Cloud agent asks for tasks to execute. The Nx Cloud service looks at the commands currently running and will see if there are any tasks that have no unfulfilled dependencies. If there are some, the Nx Cloud service will use the collected timings to create a well-size batch of tasks that it will send to the agent.

The agent sees if it has all the files required to run those tasks (`dist` folders from previous tasks). And if it doesn’t, it downloads them. When it’s done running the task, it lets the Nx Cloud service know to “unblock” other tasks in the graph. At the same time, the Nx Cloud service sends the created files and terminal outputs to the main job.
The agent sees if it has all the files required to run those tasks (`dist` folders from previous tasks). And if it doesn't, it downloads them. When it's done running the task, it lets the Nx Cloud service know to "unblock" other tasks in the graph. At the same time, the Nx Cloud service sends the created files and terminal outputs to the main job.

### Nx Cloud does not require you to rewrite the CI setup.

Expand All @@ -165,7 +165,7 @@ When using distributed task execution all the communication is done from your ag

## Summary

In this post we looked at two ways to distribute your CI: binning and using Nx Cloud’s distributed task execution.
In this post we looked at two ways to distribute your CI: binning and using Nx Cloud's distributed task execution.

Binning has been supported from Day 1 and works well for a variety of workspaces. It has drawbacks: the resource allocation, the developer ergonomics, the inability to distribute builds, and a much more complex Ci setup.

Expand Down
57 changes: 10 additions & 47 deletions ...-14-step-by-step-guide-on-creating-a-monorepo-for-react-native-apps-using-nx.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ slug: 'step-by-step-guide-on-creating-a-monorepo-for-react-native-apps-using-nx'
authors: ['Emily Xiong']
cover_image: '/blog/images/2021-10-14/92uzyqB8oJ8tZJB9wAdoWQ.png'
tags: [nx, tutorial]
description: Learn how to set up a monorepo with Nx to manage React Native mobile and React web apps with shared libraries, demonstrated through a daily horoscope app example.
---

Do you want to have both mobile and web apps in the same repo? Do you wish that you could share code between mobile and web apps? This blog post shows you how to create a React Native mobile app and a React web app in the same repo with shared libraries using Nx.
Expand Down Expand Up @@ -66,7 +67,7 @@ npm install --save @rneui/base @rneui/themed react-native-vector-icons react-nat
_**yarn add @rneui/base @rneui/themed react-native-vector-icons react-native-safe-area-context
```

In the app’s package.json at `apps/daily-horoscope-app/package.json`, under dependencies, add the above packages:
In the app's package.json at `apps/daily-horoscope-app/package.json`, under dependencies, add the above packages:

```json5
{
Expand Down Expand Up @@ -166,7 +167,7 @@ export enum AdhZodiacSign {
}
```

**Note**: the enum has a prefix “Adh” to indicate it is a model under domain “aztro-daily-horoscope”. Add this prefix to distinguish model names from component names.
**Note**: the enum has a prefix "Adh" to indicate it is a model under domain "aztro-daily-horoscope". Add this prefix to distinguish model names from component names.

This example uses icons from [Material Community Icons](https://materialdesignicons.com/). You need to create a list that contains the zodiac sign name and its matching icon.

Expand Down Expand Up @@ -542,7 +543,7 @@ npm install --save-dev redux-logger @types/redux-logger
yarn add redux-logger @types/redux-logger --dev
```

Then you need to add the redux-logger to the root store’s middleware, so the rootStore becomes:
Then you need to add the redux-logger to the root store's middleware, so the rootStore becomes:

```typescript
import logger from 'redux-logger';
Expand All @@ -559,7 +560,7 @@ const rootStore = configureStore({

Since the code is running in simulators, how to use the Redux Devtools extension and view the Redux Logger?

Open the debug menu in the simulator by entering `d` in the terminal that runs the start command. Then in the debug menu, choose “Debug with Chrome” for iOS and “Debug” for Android.
Open the debug menu in the simulator by entering `d` in the terminal that runs the start command. Then in the debug menu, choose "Debug with Chrome" for iOS and "Debug" for Android.

![](/blog/images/2021-10-14/4QVoNHRjzW0agHGnxyWvpw.avif)
_Debug Menu in iOS and Android_
Expand Down Expand Up @@ -587,7 +588,7 @@ npm install --save @react-navigation/native @react-navigation/stack react-native
yarn add @react-navigation/native @react-navigation/stack react-native-reanimated react-native-gesture-handler react-native-screens @react-native-community/masked-view
```

In the app’s package.json at `apps/daily-horoscope-app/package.json`, under dependencies, add the above packages:
In the app's package.json at `apps/daily-horoscope-app/package.json`, under dependencies, add the above packages:

```json5
{
Expand Down Expand Up @@ -694,45 +695,7 @@ export default HoroscopeCard;

Now you need to add this screen to your app. In file `apps/daily-horoscope-app/src/app/App.tsx`, you need to update it to add a stack screen for the `horoscope-card` component:

```tsx
import { store } from '@aztro-daily-horoscope/store';
import {
ZodiacSignListContainer,
HoroscopeCard,
} from '@aztro-daily-horoscope/ui';
import { NavigationContainer } from '@react-navigation/native';
import { createStackNavigator } from '@react-navigation/stack';
import * as React from 'react';
import { Provider } from 'react-redux';

const Stack = createStackNavigator();

const App = () => {
return (
<Provider store={store}>
<NavigationContainer>
<Stack.Navigator>
<Stack.Screen
name="Zodiac Sign List"
component={ZodiacSignListContainer}
/>
<Stack.Screen name="Horoscope Card" component={HoroscopeCard} />
</Stack.Navigator>
</NavigationContainer>
</Provider>
);
};

export default App;
```

In the `libs/ui/src/lib/zodiac-sign-list/zodiac-sign-list.tsx`, you need to trigger navigation when the list item got pressed.

Below code uses [`useNavigation`](https://reactnavigation.org/docs/use-navigation/) hook from [React Navigation](https://reactnavigation.org/) library. When the list item got pressed, it is going to call `navigation.navigate(‘Horoscope Card’)` to navigate the `horoscope-card` component you created above.

[https://gist.github.com/xiongemi/c78c719e70aa4948b98e68033d7fe4a3](https://gist.github.com/xiongemi/c78c719e70aa4948b98e68033d7fe4a3)

```tsx {% fileName="App.tsx" %}
```tsx {% fileName="zodiac-sign-list.tsx" %}
import {
AdhZodiacSignItem,
AdhZodiacSignList,
Expand Down Expand Up @@ -800,7 +763,7 @@ nx generate lib services
In the services folder, add the below files:

- `aztro-horoscope-response.interface.ts` defines what the response object looks like. It has a transform function to transform response data to the app domain model.
- `aztro.service.ts` calls the API to get the user’s horoscope based on the zodiac sign and day.
- `aztro.service.ts` calls the API to get the user's horoscope based on the zodiac sign and day.

```typescript {% fileName="aztro-horoscope-response.interface.ts" %}
import { AdhHoroscope, AdhZodiacSign } from '@aztro-daily-horoscope/models';
Expand Down Expand Up @@ -899,7 +862,7 @@ export interface HoroscopeState {

- `loadingStatus` is the API request status from `aztro.service`.
- `error` is the API request error from `aztro.service`.
- `zodiacSignItem` is the user’s selected zodiac sign.
- `zodiacSignItem` is the user's selected zodiac sign.
- `day` is the parameter passed to `aztro.service`.
- `horoscope` is the transformed response from `aztro.service.`

Expand Down Expand Up @@ -1255,7 +1218,7 @@ AppRegistry.runApplication('main', {
});
```

Copy your code from `daily-horoscope-app`’s app file to `daily-horoscope-web`’s app file and add styles for the icon font files:
Copy your code from `daily-horoscope-app`'s app file to `daily-horoscope-web`'s app file and add styles for the icon font files:

```tsx {% fileName="app.tsx" %}
import { rootStore } from '@aztro-daily-horoscope/store';
Expand Down
Loading