···11-# LunAST
22-33-LunAST is an experimental in-development [AST](https://en.wikipedia.org/wiki/Abstract_syntax_tree)-based remapper and patcher for Webpack modules.
44-55-## Introduction
66-77-Modern Webpack patching functions off of matching existing minified code (using a string or regular expression) and then replacing it. While this is an easy to use and powerful way of patching Webpack modules, there are many downsides:
88-99-- Even the smallest change can break patches, which can require lots of maintenance, especially on large Discord bundler changes.
1010-- Fetching exports from a Webpack module will sometimes result in minified export names. These exports must be manually remapped to human readable names by a library extension.
1111-- Making complicated patches is extremely difficult and means your patch has more points of failure.
1212-1313-To solve this, LunAST generates [the ESTree format](https://github.com/estree/estree) with a handful of libraries ([meriyah](https://github.com/meriyah/meriyah), [estree-toolkit](https://github.com/sarsamurmu/estree-toolkit), [astring](https://github.com/davidbonnet/astring)) on each Webpack module. This makes large-scale manipulation and mapping feasible, by allowing you to write code to detect what modules you want to find.
1414-1515-## Usage
1616-1717-### Embedding into your own code
1818-1919-LunAST is not ready to be used in other projects just yet. In the future, LunAST will be a standalone library.
2020-2121-### Registering a processor
2222-2323-LunAST functions off of "processors". Processors have a unique ID, an optional filter (string or regex) on what to parse, and a process function which receives the AST.
2424-2525-The process function returns a boolean, which when true will unregister the processor. Once you have found what you're looking for, you can return true to skip parsing any other subsequent module, speeding up load times.
2626-2727-LunAST includes some core processors, and extensions can register their own processors (citation needed).
2828-2929-```ts
3030-register({
3131- name: "UniqueIDForTheProcessorSystem",
3232- find: "some string or regex to search for", // optional
3333- priority: 0, // optional
3434- process({ id, ast, lunast }) {
3535- // do some stuff with the ast
3636- return false; // return true to unregister
3737- }
3838-});
3939-```
4040-4141-### Mapping with LunAST
4242-4343-LunAST can use proxies to remap minified export names to more human readable ones. Let's say that you determined the module ID and export name of a component you want in a module.
4444-4545-First, you must define the type. A type contains a unique name and a list of fields. These fields contain the minified name and the human-readable name that can be used in code.
4646-4747-Then, register the module, with the ID passed to you in the process function. Specify its type so the remapper knows what fields to remap. It is suggested to name the module and type with the same name.
4848-4949-```ts
5050-process({ id, ast, lunast }) {
5151- let exportName: string | null = null;
5252-5353- // some code to discover the export name...
5454-5555- if (exportName != null) {
5656- lunast.addType({
5757- name: "SomeModule",
5858- fields: [
5959- {
6060- name: "SomeComponent",
6161- unmapped: exportName
6262- }
6363- ]
6464- });
6565- lunast.addModule({
6666- name: "SomeModule",
6767- id,
6868- type: "SomeModule"
6969- });
7070- return true;
7171- }
7272-7373- return false;
7474-}
7575-```
7676-7777-Then, you need to specify the type of the module in `types.ts`. Using the `import` statement in Webpack modules is not supported yet. Hopefully this step is automated in the future.
7878-7979-After all this, fetch the remapped module and its remapped field:
8080-8181-```ts
8282-moonlight.lunast.remap("SomeModule").SomeComponent
8383-```
8484-8585-### Patching with LunAST
8686-8787-LunAST also enables you to modify the AST and then rebuild a module string from the modified AST. It is suggested you read the [estree-toolkit](https://estree-toolkit.netlify.app/welcome) documentation.
8888-8989-You can use the `magicAST` function to turn some JavaScript code into another AST node, and then merge/replace the original AST.
9090-9191-**After you modify the AST, call the markDirty function.** LunAST will not know to replace the module otherwise.
9292-9393-```ts
9494-process({ ast, markDirty }) {
9595- const node = /* do something with the AST */;
9696- if (node != null) {
9797- const replacement = magicAST("return 1 + 1");
9898- node.replaceWith(replacement);
9999- markDirty();
100100- return true;
101101- }
102102-103103- return false;
104104-}
105105-```
106106-107107-## FAQ
108108-109109-### How do you fetch the scripts to parse?
110110-111111-Fetching the content of the `<script>` tags is impossible, and making a `fetch` request would result in different headers to what the client would normally send. We use `Function.prototype.toString()` and wrap the function in parentheses to ensure the anonymous function is valid JavaScript.
112112-113113-### Isn't this slow?
114114-115115-Not really. LunAST runs in roughly ~10ms on [my](https://github.com/NotNite) machine, with filtering for what modules to parse. Parsing every module takes only a second. There are future plans to cache and parallelize the process, so that load times are only slow once.
116116-117117-You can measure how long LunAST took to process with the `moonlight.lunast.elapsed` variable.
118118-119119-### Does this mean patches are dead?
120120-121121-No. Patches will continue to serve their purpose and be supported in moonlight, no matter what. LunAST should also work with patches, but patches may conflict or not match.
122122-123123-[astring](https://github.com/davidbonnet/astring) may need to be forked in the future to output code without whitespace, in the event patches fail to match on AST-patched code.
124124-125125-### This API surface seems kind of bad
126126-127127-This is still in heavy development and all suggestions on how to improve it are welcome. :3
128128-129129-### Can I help?
130130-131131-Discussion takes place in the [moonlight Discord server](https://discord.gg/FdZBTFCP6F) and its `#lunast-devel` channel.
-22
packages/lunast/TODO.md
···11-# LunAST TODO
22-33-- [ ] Experiment more! We need to know what's bad with this
44-- [ ] Write utility functions for imports, exports, etc.
55- - [ ] Imports
66- - [x] Exports
77- - [ ] Constant bindings for an object
88-- [ ] Map Z/ZP to default
99-- [x] Steal Webpack require and use it in our LunAST instance
1010-- [ ] Map `import` statements to LunAST
1111-- [x] Support patching in the AST
1212- - Let user modify the AST, have a function to flag it as modified, if it's modified we serialize it back into a string and put it back into Webpack
1313- - We already have a `priority` system for this
1414-- [ ] Run in parallel with service workers
1515- - This is gonna require making Webpack entrypoint async and us doing kickoff ourselves
1616-- [ ] Support lazy loaded chunks
1717- - Works right now, but will break when caching is implemented
1818-- [ ] Split into a new repo on GitHub, publish to NPM maybe
1919-- [ ] Implement caching based off of the client build and LunAST commit
2020- - Means you only have to have a long client start once per client build
2121-- [ ] Process in CI to use if available on startup
2222- - Should mean, if you're lucky, client starts only take the extra time to make the request
···11-import { RemapField, RemapModule, RemapType } from "./types";
22-import { Remapped } from "./modules";
33-import { getProcessors, parseFixed } from "./utils";
44-import { Processor, ProcessorState } from "./remap";
55-import { generate } from "astring";
66-77-export default class LunAST {
88- private modules: Record<string, RemapModule>;
99- private types: Record<string, RemapType>;
1010- private successful: Set<string>;
1111-1212- private typeCache: Record<string, RemapType | null>;
1313- private fieldCache: Record<
1414- string,
1515- Record<string | symbol, RemapField | null>
1616- >;
1717- private processors: Processor[];
1818- private defaultRequire?: (id: string) => any;
1919- private getModuleSource?: (id: string) => string;
2020-2121- elapsed: number;
2222-2323- constructor() {
2424- this.modules = {};
2525- this.types = {};
2626- this.successful = new Set();
2727-2828- this.typeCache = {};
2929- this.fieldCache = {};
3030- this.processors = getProcessors();
3131-3232- this.elapsed = 0;
3333- }
3434-3535- public static getVersion() {
3636- // TODO: embed version in build when we move this to a new repo
3737- // this is here for caching based off of the lunast commit ID
3838- return "dev";
3939- }
4040-4141- public parseScript(id: string, code: string): Record<string, string> {
4242- const start = performance.now();
4343-4444- const available = [...this.processors]
4545- .sort((a, b) => (a.priority ?? 0) - (b.priority ?? 0))
4646- .filter((x) => {
4747- if (x.find == null) return true;
4848- const finds = Array.isArray(x.find) ? x.find : [x.find];
4949- return finds.every((find) =>
5050- typeof find === "string" ? code.indexOf(find) !== -1 : find.test(code)
5151- );
5252- })
5353- .filter((x) => x.manual !== true);
5454-5555- const ret = this.parseScriptInternal(id, code, available);
5656-5757- const end = performance.now();
5858- this.elapsed += end - start;
5959-6060- return ret;
6161- }
6262-6363- // This is like this so processors can trigger other processors while they're parsing
6464- private parseScriptInternal(
6565- id: string,
6666- code: string,
6767- processors: Processor[]
6868- ) {
6969- const ret: Record<string, string> = {};
7070- if (processors.length === 0) return ret;
7171-7272- // Wrap so the anonymous function is valid JS
7373- const module = parseFixed(`(\n${code}\n)`);
7474- let dirty = false;
7575- const state: ProcessorState = {
7676- id,
7777- ast: module,
7878- lunast: this,
7979- markDirty: () => {
8080- dirty = true;
8181- },
8282- trigger: (id, tag) => {
8383- const source = this.getModuleSourceById(id);
8484- if (source == null) return;
8585- if (this.successful.has(tag)) return;
8686- const processor = this.processors.find((x) => x.name === tag);
8787- if (processor == null) return;
8888- const theirRet = this.parseScriptInternal(id, source, [processor]);
8989- Object.assign(ret, theirRet);
9090- }
9191- };
9292-9393- for (const processor of processors) {
9494- if (processor.process(state)) {
9595- this.processors.splice(this.processors.indexOf(processor), 1);
9696- this.successful.add(processor.name);
9797- }
9898- }
9999-100100- const str = dirty ? generate(module) : null;
101101- if (str != null) ret[id] = str;
102102-103103- return ret;
104104- }
105105-106106- public getType(name: string) {
107107- return (
108108- this.typeCache[name] ?? (this.typeCache[name] = this.types[name] ?? null)
109109- );
110110- }
111111-112112- public getIdForModule(name: string) {
113113- return Object.values(this.modules).find((x) => x.name === name)?.id ?? null;
114114- }
115115-116116- public addModule(module: RemapModule) {
117117- if (!this.modules[module.name]) {
118118- this.modules[module.name] = module;
119119- } else {
120120- throw new Error(
121121- `Module ${module.name} already registered (${
122122- this.modules[module.name].id
123123- })`
124124- );
125125- }
126126- }
127127-128128- public addType(type: RemapType) {
129129- if (!this.types[type.name]) {
130130- this.types[type.name] = type;
131131- } else {
132132- throw new Error(`Type ${type.name} already registered`);
133133- }
134134- }
135135-136136- public proxy(obj: any, type: RemapType): any {
137137- const fields =
138138- this.fieldCache[type.name] ?? (this.fieldCache[type.name] = {});
139139-140140- return new Proxy(obj, {
141141- get: (target, prop) => {
142142- const field =
143143- fields[prop] ??
144144- (fields[prop] = type.fields.find((x) => x.name === prop) ?? null);
145145- if (field) {
146146- const fieldType =
147147- field.type != null ? this.getType(field.type) : null;
148148- const name = field.unmapped ?? field.name;
149149- if (fieldType != null) {
150150- return this.proxy(target[name], fieldType);
151151- } else {
152152- return target[name];
153153- }
154154- } else {
155155- return target[prop];
156156- }
157157- }
158158- });
159159- }
160160-161161- // TODO: call this with require we obtain from the webpack entrypoint
162162- public setDefaultRequire(require: (id: string) => any) {
163163- this.defaultRequire = require;
164164- }
165165-166166- public setModuleSourceGetter(getSource: (id: string) => string) {
167167- this.getModuleSource = getSource;
168168- }
169169-170170- public getModuleSourceById(id: string) {
171171- return this.getModuleSource?.(id) ?? null;
172172- }
173173-174174- public remap<Id extends keyof Remapped>(
175175- id: Id,
176176- require?: (id: string) => any
177177- ): Remapped[Id] | null {
178178- const mappedModule = this.modules[id];
179179- if (!mappedModule) return null;
180180-181181- const realRequire = require ?? this.defaultRequire;
182182- if (!realRequire) return null;
183183-184184- const module = realRequire(mappedModule.id);
185185- if (module == null) return null;
186186-187187- const type = this.getType(mappedModule.type);
188188- if (type != null) {
189189- return this.proxy(module, type);
190190- } else {
191191- return module;
192192- }
193193- }
194194-}
195195-196196-export { Remapped } from "./modules";
-4
packages/lunast/src/modules.ts
···11-// This kinda sucks, TODO figure out a better way to do this dynamically
22-import "./modules/test";
33-44-export type Remapped = Record<string, never>;
···11-import type LunAST from ".";
22-import type { Program } from "estree-toolkit/dist/generated/types";
33-44-export type Processor = {
55- name: string;
66- find?: (string | RegExp)[] | (string | RegExp);
77- priority?: number;
88- manual?: boolean;
99- process: (state: ProcessorState) => boolean;
1010-};
1111-export type ProcessorState = {
1212- id: string;
1313- ast: Program;
1414- lunast: LunAST;
1515- markDirty: () => void;
1616- trigger: (id: string, tag: string) => void;
1717-};
-16
packages/lunast/src/types.ts
···11-export type RemapModule = {
22- name: string; // the name you require it by in your code
33- id: string; // the resolved webpack module ID (usually a number)
44- type: string;
55-};
66-77-export type RemapType = {
88- name: string;
99- fields: RemapField[];
1010-};
1111-1212-export type RemapField = {
1313- name: string; // the name of the field in the proxy (human readable)
1414- unmapped?: string; // the name of the field in discord source (minified)
1515- type?: string;
1616-};
-211
packages/lunast/src/utils.ts
···11-import type { Processor } from "./remap";
22-import { traverse, is, Scope, Binding, NodePath } from "estree-toolkit";
33-// FIXME something's fishy with these types
44-import type {
55- Expression,
66- ExpressionStatement,
77- ObjectExpression,
88- Program,
99- ReturnStatement
1010-} from "estree-toolkit/dist/generated/types";
1111-import { parse } from "meriyah";
1212-1313-export const processors: Processor[] = [];
1414-1515-export function register(processor: Processor) {
1616- processors.push(processor);
1717-}
1818-1919-export function getProcessors() {
2020- // Clone the array to prevent mutation
2121- return [...processors];
2222-}
2323-2424-export type ExpressionWithScope = {
2525- expression: Expression;
2626- scope: Scope;
2727-};
2828-2929-function getParent(path: NodePath) {
3030- let parent = path.parentPath;
3131- while (!is.program(parent)) {
3232- parent = parent?.parentPath ?? null;
3333- if (
3434- parent == null ||
3535- parent.node == null ||
3636- ![
3737- "FunctionExpression",
3838- "ExpressionStatement",
3939- "CallExpression",
4040- "Program"
4141- ].includes(parent.node.type)
4242- ) {
4343- return null;
4444- }
4545- }
4646-4747- if (!is.functionExpression(path.parent)) return null;
4848- return path.parent;
4949-}
5050-5151-export function getExports(ast: Program) {
5252- const ret: Record<string, ExpressionWithScope> = {};
5353-5454- traverse(ast, {
5555- $: { scope: true },
5656- BlockStatement(path) {
5757- if (path.scope == null) return;
5858- const parent = getParent(path);
5959- if (parent == null) return;
6060-6161- for (let i = 0; i < parent.params.length; i++) {
6262- const param = parent.params[i];
6363- if (!is.identifier(param)) continue;
6464- const binding: Binding | undefined = path.scope!.getBinding(param.name);
6565- if (!binding) continue;
6666-6767- // module
6868- if (i === 0) {
6969- for (const reference of binding.references) {
7070- if (!is.identifier(reference.node)) continue;
7171- if (!is.assignmentExpression(reference.parentPath?.parentPath))
7272- continue;
7373-7474- const exportsNode = reference.parentPath?.parentPath.node;
7575- if (!is.memberExpression(exportsNode?.left)) continue;
7676- if (!is.identifier(exportsNode.left.property)) continue;
7777- if (exportsNode.left.property.name !== "exports") continue;
7878-7979- const exports = exportsNode?.right;
8080- if (!is.objectExpression(exports)) continue;
8181-8282- for (const property of exports.properties) {
8383- if (!is.property(property)) continue;
8484- if (!is.identifier(property.key)) continue;
8585- if (!is.expression(property.value)) continue;
8686- ret[property.key.name] = {
8787- expression: property.value,
8888- scope: path.scope
8989- };
9090- }
9191- }
9292- }
9393- // TODO: exports
9494- else if (i === 1) {
9595- for (const reference of binding.references) {
9696- if (!is.identifier(reference.node)) continue;
9797- if (reference.parentPath == null) continue;
9898- if (!is.memberExpression(reference.parentPath.node)) continue;
9999- if (!is.identifier(reference.parentPath.node.property)) continue;
100100-101101- const assignmentExpression = reference.parentPath.parentPath?.node;
102102- if (!is.assignmentExpression(assignmentExpression)) continue;
103103-104104- ret[reference.parentPath.node.property.name] = {
105105- expression: assignmentExpression.right,
106106- scope: path.scope
107107- };
108108- }
109109- }
110110- }
111111- }
112112- });
113113-114114- return ret;
115115-}
116116-117117-// TODO: util function to resolve the value of an expression
118118-export function getPropertyGetters(ast: Program) {
119119- const ret: Record<string, ExpressionWithScope> = {};
120120-121121- traverse(ast, {
122122- $: { scope: true },
123123- CallExpression(path) {
124124- if (path.scope == null) return;
125125- if (!is.callExpression(path.node)) return;
126126- if (!is.memberExpression(path.node.callee)) return;
127127- if (!is.identifier(path.node?.callee?.property)) return;
128128- if (path.node.callee.property.name !== "d") return;
129129-130130- const arg = path.node.arguments.find((node): node is ObjectExpression =>
131131- is.objectExpression(node)
132132- );
133133- if (!arg) return;
134134-135135- for (const property of arg.properties) {
136136- if (!is.property(property)) continue;
137137- if (!is.identifier(property.key)) continue;
138138- if (!is.functionExpression(property.value)) continue;
139139- if (!is.blockStatement(property.value.body)) continue;
140140-141141- const returnStatement = property.value.body.body.find(
142142- (node): node is ReturnStatement => is.returnStatement(node)
143143- );
144144- if (!returnStatement || !returnStatement.argument) continue;
145145- ret[property.key.name] = {
146146- expression: returnStatement.argument,
147147- scope: path.scope
148148- };
149149- }
150150-151151- this.stop();
152152- }
153153- });
154154-155155- return ret;
156156-}
157157-158158-// The ESTree types are mismatched with estree-toolkit, but ESTree is a standard so this is fine
159159-export function parseFixed(code: string): Program {
160160- return parse(code) as any as Program;
161161-}
162162-163163-export function magicAST(code: string) {
164164- // Wraps code in an IIFE so you can type `return` and all that goodies
165165- // Might not work for some other syntax issues but oh well
166166- const tree = parse("(()=>{" + code + "})()");
167167-168168- const expressionStatement = tree.body[0] as ExpressionStatement;
169169- if (!is.expressionStatement(expressionStatement)) return null;
170170- if (!is.callExpression(expressionStatement.expression)) return null;
171171- if (!is.arrowFunctionExpression(expressionStatement.expression.callee))
172172- return null;
173173- if (!is.blockStatement(expressionStatement.expression.callee.body))
174174- return null;
175175- return expressionStatement.expression.callee.body;
176176-}
177177-178178-export function getImports(ast: Program) {
179179- const ret: Record<string, ExpressionWithScope> = {};
180180-181181- traverse(ast, {
182182- $: { scope: true },
183183- BlockStatement(path) {
184184- if (path.scope == null) return;
185185- const parent = getParent(path);
186186- if (parent == null) return;
187187-188188- const require = parent.params[2];
189189- if (!is.identifier(require)) return;
190190- const references = path.scope.getOwnBinding(require.name)?.references;
191191- if (references == null) return;
192192- for (const reference of references) {
193193- if (!is.callExpression(reference.parentPath)) continue;
194194- if (reference.parentPath.node?.arguments.length !== 1) continue;
195195- if (!is.variableDeclarator(reference.parentPath.parentPath)) continue;
196196- if (!is.identifier(reference.parentPath.parentPath.node?.id)) continue;
197197-198198- const moduleId = reference.parentPath.node.arguments[0];
199199- if (!is.literal(moduleId)) continue;
200200- if (moduleId.value == null) continue;
201201-202202- ret[moduleId.value.toString()] = {
203203- expression: reference.parentPath.parentPath.node.id,
204204- scope: path.scope
205205- };
206206- }
207207- }
208208- });
209209-210210- return ret;
211211-}
···88} from "./extension";
99import type EventEmitter from "events";
1010import type LunAST from "@moonlight-mod/lunast";
1111+import type Moonmap from "@moonlight-mod/moonmap";
11121213export type MoonlightHost = {
1314 asarPath: string;
···4647 getNatives: (ext: string) => any | undefined;
4748 getLogger: (id: string) => Logger;
4849 lunast: LunAST;
5050+ moonmap: Moonmap;
4951};
50525153export enum MoonlightEnv {
+3
packages/types/src/index.ts
···1919export * from "./logger";
2020export * as constants from "./constants";
21212222+export type { AST } from "@moonlight-mod/lunast";
2323+export { ModuleExport, ModuleExportType } from "@moonlight-mod/moonmap";
2424+2225declare global {
2326 const MOONLIGHT_ENV: MoonlightEnv;
2427 const MOONLIGHT_PROD: boolean;