Skip to content

Commit

Permalink
unconflicted
Browse files Browse the repository at this point in the history
  • Loading branch information
peterbe committed Jul 29, 2020
2 parents e8db01d dde924c commit 1fa0b76
Show file tree
Hide file tree
Showing 108 changed files with 5,374 additions and 6,685 deletions.
12 changes: 1 addition & 11 deletions .env-dist
Original file line number Diff line number Diff line change
@@ -1,15 1,5 @@
#BUILD_ARCHIVE_ROOT=archivecontent/files
#CONTENT_ARCHIVE_ROOT=archivecontent/files
#BUILD_STUMTPTOWN_ROOT=stumptown/packaged

# To build only a subset of documents (which is much faster) you can
# use command line options with the cli. But if you don't want to type
# --locales=en-us every time, you can set:
#BUILD_LOCALES=en-us
# If you don't want to type --foldersearch=^web every time you can set:
#BUILD_FOLDERSEARCHES=^web

# Note that all these filters are *union* meaning
# `BUILD_LOCALES=en-us,sv-se` for example will match 'en-us' OR 'sv-se'

# See documentation in docs/envvars.md for more information about this
#BUILD_FLAW_LEVELS=broken_links:error, macros:ignore
10 changes: 8 additions & 2 deletions .github/workflows/pr-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 13,11 @@ name: PR Build

on:
pull_request:
branches: [master]
branches:
- master
# These line is temporary! At least until we're done using Gregor's
# reductio-reloaded branch instead of mdn.
- reductio-reloaded

jobs:
build:
Expand Down Expand Up @@ -78,4 82,6 @@ jobs:
# functional tests have a gap in them but actually building
# real content exposes it.
node content build --no-progressbar --files="${{ env.GIT_DIFF }}"
export BUILD_NO_PROGRESSBAR=true
export BUILD_FILES="${{ env.GIT_DIFF }}"
yarn build
6 changes: 5 additions & 1 deletion .github/workflows/testing.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 7,11 @@ on:
push:
branches: [master]
pull_request:
branches: [master]
branches:
- master
# These line is temporary! At least until we're done using Gregor's
# reductio-reloaded branch instead of mdn.
- reductio-reloaded

jobs:
build:
Expand Down
19 changes: 5 additions & 14 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,16 1,13 @@
# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.

# dependencies
/node_modules
node_modules/
/.pnp
.pnp.js

# testing
/coverage

# production
/build

# misc
.DS_Store
.env.local
Expand All @@ -22,21 19,15 @@
npm-debug.log*
yarn-debug.log*
yarn-error.log*
/server/node_modules/
/client/node_modules/
/client/build/
/server/dist/
/ssr/node_modules/
/ssr/dist/
/watcher/node_modules/
/content/node_modules/
/content/build/
/content/_all-titles.json
.next/
titles.json
# Remove this once we're using `/testing/content/` fully.
/ci-content/
/kumascript/coverage/
/kumascript/node_modules/
/testing/node_modules/
/client/build/
/ssr/dist/

# Can be removed once content lives in its own repo
/archivecontent/files
Expand Down
54 changes: 7 additions & 47 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 15,7 @@ To run Yari locally, you'll first need to install [git](https://git-scm.com/), [
open http://localhost:3000

If you prefer, you can fork the repo first and do the `git clone` with
*your* fork instead of the `mdn` one.
_your_ fork instead of the `mdn` one.

### How to stay up-to-date

Expand All @@ -24,7 24,6 @@ up-to-date with these commands:

git pull origin master
yarn
yarn clean
yarn start

These are also good steps to always take when you embark on making a change.
Expand Down Expand Up @@ -56,13 55,7 @@ might expand with more documentation specifically for contributing to the
content exclusively.

The `yarn start` command encapsulates the front-end dev server
(on `localhost:3000`) and the `server` (on `localhost:5000`)
as well as the `watcher`. The `watcher` triggers a build when a
file changes (or is added!).
The `yarn start` command also first runs a command that gathers up *all*
the document URLs and their titles (plus some other metadata). This can
take a while but it's cached to disk and is automatically invalidated if
any of the source code changes.
(on `localhost:3000`) and the `server` (on `localhost:5000`).

All the sub-commands of `yarn start` can be broken down and run individually
if you want to work more rapidly.
Expand All @@ -87,15 80,6 @@ The `server` has two main jobs:
1. Simulate serving the site (e.g. from a server, S3 or a CDN).
2. Trigger builds of documents that haven't been built, by URL.

If you don't use the server you can manually use the CLI to build a page.
For example:

node content build -l en-us -f web/html/element

...will build all documents that match that folder prefix. But you don't
need to do that up front since the server will "fill in the gaps" by
triggering a build if you simply try to view it.

### Linting

All JavaScript and TypeScript code needs to be formatted with `prettier`
Expand Down Expand Up @@ -182,38 166,14 @@ in advance. One potential advantage is that you can get a more complete
list of all possible "flaws" across all documents before you even visit them.
The most fundamental CLI command is:

yarn prebuild # if you haven't run 'yarn start' recently
node content build

You can use `--help` to find various options such as filtering. For example,
to build all `en-US` documents under the `Web/CSS/` subtree, you can run this:

node content build -l en-us -f web/css

The files get put into the `client/build/` and that folder should be ready
to be uploaded to static file hosting. Note the `-f` (aka `--foldersearch`)
is case-insensitive.
yarn build

### What gets built

Every `index.html` becomes three files:

* `index.html` fully formed and complete HTML file
* `index.json` the React needed state to build the page in the client
* `index.hash` a short hash digest of the cache key used to build this page

### Caching

When building, we attempt to use a disk-based cache. It computes a hash
for every document by combining...

* The content of the `index.html` file.
* The combined source code of all `.js`, `.tsx`, and also all `package.json`
and `yarn.lock`.
Every `index.html` becomes two files:

If you run `node content build -l en-us -f web/css` *twice* you'll find it
runs *much* faster the second time. But any little change to any source
code file should invalidate the cache.
- `index.html` fully formed and complete HTML file
- `index.json` the React needed state to build the page in the client

### Flaw checks

Expand All @@ -223,7 183,7 @@ severe but they should never block a full build.

More information about how to set flaws can be found in `docs/envvars.md`.

Essentially, the default is to *warn* about any flaw and you can see
Essentially, the default is to _warn_ about any flaw and you can see
those flaws when using `http://localhost:3000`. But for completed builds,
all flaws are ignored. This makes the build faster and there's also
no good place to display the flaws in a production-grade build.
Expand Down
114 changes: 114 additions & 0 deletions build/build-options.js
Original file line number Diff line number Diff line change
@@ -0,0 1,114 @@
require("dotenv").config();

const {
FLAW_LEVELS,
DEFAULT_FLAW_LEVELS,
VALID_FLAW_CHECKS,
FILES,
FOLDERSEARCH,
NO_PROGRESSBAR,
FIX_FLAWS,
FIX_FLAWS_DRY_RUN,
FIX_FLAWS_VERBOSE,
} = require("./constants");

const options = Object.freeze({
flawLevels: parseFlawLevels(DEFAULT_FLAW_LEVELS),
files: parseFiles(FILES),
folderSearch: parseFoldersearch(FOLDERSEARCH),
noProgressbar: NO_PROGRESSBAR,
fixFlaws: FIX_FLAWS,
fixFlawsDryRun: FIX_FLAWS_DRY_RUN,
fixFlawsVerbose: FIX_FLAWS_VERBOSE,
});

function parseFiles(filesStringList) {
// The get-diff-action, which we use in the "PR Builds" CI,
// will make this a massive string that looks like
// this: `'content/files/en-us/a/index.html','content/files/en-us/a/index.html'`
// so we need to turn that into an array:
// ["content/files/en-us/a/index.html", "content/files/en-us/b/index.html"]`
// Note, when you use get-diff-action in GitHub Actions, it's a comma
// but if you use the manualy `git diff --name-only ...` on your command
// line it's a newline.
return new Set(
filesStringList
.split(/[,\n]/)
.map((item) => {
// Remove any single or double-quote bookends.
return item.replace(/^(['"])(.*)\1$/, "$2");
})
.filter((s) => s)
);
}

function parseFoldersearch(searchpattern) {
if (searchpattern) {
// TODO: Consider turning it into a regex if there are * or $ or ^ in it
return searchpattern.toLowerCase();
}
return null;
}

// Override based on env vars but only for options that are *not*
// exclusive to building everyhing.
function parseFlawLevels(flawChecks) {
const checks = flawChecks
.split(",")
.map((x) => x.trim())
.filter((x) => x)
.map((x) => x.split(":").map((s) => s.trim()));

// Check that it doesn't contain more than 1 wildcard,
// because that'd make no sense.
const wildcards = checks.filter((tuple) => tuple[0] === "*");
if (wildcards.length > 1) {
throw new Error(`Can only be 1 wild card (not: ${wildcards})`);
}

// Put any wildcards (e.g. '*:warn') first
checks.sort((a, b) => {
if (a[0] === "*" && b[0] !== "*") {
return -1;
} else if (a[0] !== "*" && b[0] === "*") {
return 1;
}
return 0;
});

const checked = new Map();

// Unless specified, set 'ignore' on all of them first.
for (const check of VALID_FLAW_CHECKS) {
checked.set(check, FLAW_LEVELS.IGNORE);
}

const levelValues = Object.values(FLAW_LEVELS);

for (const tuple of checks) {
if (tuple.length !== 2) {
throw new Error(`Not a tuple pair of 2 (${tuple})`);
}
const [identifier, level] = tuple;
if (!levelValues.includes(level)) {
throw new Error(`Invalid level: '${level}' (only ${levelValues})`);
}
if (identifier === "*") {
for (const check of VALID_FLAW_CHECKS) {
checked.set(check, level);
}
} else if (!VALID_FLAW_CHECKS.has(identifier)) {
throw new Error(
`Unrecognized flaw identifier: '${identifier}' (only ${[
...VALID_FLAW_CHECKS,
]})`
);
} else {
checked.set(identifier, level);
}
}

return checked;
}

module.exports = options;
Loading

0 comments on commit 1fa0b76

Please sign in to comment.