notnite.com
this repo has no description
LunAST: Add README
+154
-8
+131
packages/lunast/README.md
+131
packages/lunast/README.md
···
1
+
# LunAST
2
+
3
+
LunAST is an experimental in-development [AST](https://en.wikipedia.org/wiki/Abstract_syntax_tree)-based remapper and patcher for Webpack modules.
4
+
5
+
## Introduction
6
+
7
+
Modern Webpack patching functions off of matching existing minified code (using a string or regular expression) and then replacing it. While this is an easy to use and powerful way of patching Webpack modules, there are many downsides:
8
+
9
+
- Even the smallest change can break patches, which can require lots of maintenance, especially on large Discord bundler changes.
10
+
- Fetching exports from a Webpack module will sometimes result in minified export names. These exports must be manually remapped to human readable names by a library extension.
11
+
- Making complicated patches is extremely difficult and means your patch has more points of failure.
12
+
13
+
To solve this, LunAST generates [the ESTree format](https://github.com/estree/estree) with a handful of libraries ([meriyah](https://github.com/meriyah/meriyah), [estree-toolkit](https://github.com/sarsamurmu/estree-toolkit), [astring](https://github.com/davidbonnet/astring)) on each Webpack module. This makes large-scale manipulation and mapping feasible, by allowing you to write code to detect what modules you want to find.
14
+
15
+
## Usage
16
+
17
+
### Embedding into your own code
18
+
19
+
LunAST is not ready to be used in other projects just yet. In the future, LunAST will be a standalone library.
20
+
21
+
### Registering a processor
22
+
23
+
LunAST functions off of "processors". Processors have a unique ID, an optional filter (string or regex) on what to parse, and a process function which receives the AST.
24
+
25
+
The process function returns a boolean, which when true will unregister the processor. Once you have found what you're looking for, you can return true to skip parsing any other subsequent module, speeding up load times.
26
+
27
+
LunAST includes some core processors, and extensions can register their own processors (citation needed).
28
+
29
+
```ts
30
+
register({
31
+
name: "UniqueIDForTheProcessorSystem",
32
+
find: "some string or regex to search for", // optional
33
+
priority: 0, // optional
34
+
process({ id, ast, lunast }) {
35
+
// do some stuff with the ast
36
+
return false; // return true to unregister
37
+
}
38
+
});
39
+
```
40
+
41
+
### Mapping with LunAST
42
+
43
+
LunAST can use proxies to remap minified export names to more human readable ones. Let's say that you determined the module ID and export name of a component you want in a module.
44
+
45
+
First, you must define the type. A type contains a unique name and a list of fields. These fields contain the minified name and the human-readable name that can be used in code.
46
+
47
+
Then, register the module, with the ID passed to you in the process function. Specify its type so the remapper knows what fields to remap. It is suggested to name the module and type with the same name.
48
+
49
+
```ts
50
+
process({ id, ast, lunast }) {
51
+
let exportName: string | null = null;
52
+
53
+
// some code to discover the export name...
54
+
55
+
if (exportName != null) {
56
+
lunast.addType({
57
+
name: "SomeModule",
58
+
fields: [
59
+
{
60
+
name: "SomeComponent",
61
+
unmapped: exportName
62
+
}
63
+
]
64
+
});
65
+
lunast.addModule({
66
+
name: "SomeModule",
67
+
id,
68
+
type: "SomeModule"
69
+
});
70
+
return true;
71
+
}
72
+
73
+
return false;
74
+
}
75
+
```
76
+
77
+
Then, you need to specify the type of the module in `types.ts`. Using the `import` statement in Webpack modules is not supported yet. Hopefully this step is automated in the future.
78
+
79
+
After all this, fetch the remapped module and its remapped field:
80
+
81
+
```ts
82
+
moonlight.lunast.remap("SomeModule").SomeComponent
83
+
```
84
+
85
+
### Patching with LunAST
86
+
87
+
LunAST also enables you to modify the AST and then rebuild a module string from the modified AST. It is suggested you read the [estree-toolkit](https://estree-toolkit.netlify.app/welcome) documentation.
88
+
89
+
You can use the `magicAST` function to turn some JavaScript code into another AST node, and then merge/replace the original AST.
90
+
91
+
**After you modify the AST, call the markDirty function.** LunAST will not know to replace the module otherwise.
92
+
93
+
```ts
94
+
process({ ast, markDirty }) {
95
+
const node = /* do something with the AST */;
96
+
if (node != null) {
97
+
const replacement = magicAST("return 1 + 1");
98
+
node.replaceWith(replacement);
99
+
markDirty();
100
+
return true;
101
+
}
102
+
103
+
return false;
104
+
}
105
+
```
106
+
107
+
## FAQ
108
+
109
+
### How do you fetch the scripts to parse?
110
+
111
+
Fetching the content of the `<script>` tags is impossible, and making a `fetch` request would result in different headers to what the client would normally send. We use `Function.prototype.toString()` and wrap the function in parentheses to ensure the anonymous function is valid JavaScript.
112
+
113
+
### Isn't this slow?
114
+
115
+
Not really. LunAST runs in roughly ~10ms on [my](https://github.com/NotNite) machine, with filtering for what modules to parse. Parsing every module takes only a second. There are future plans to cache and parallelize the process, so that load times are only slow once.
116
+
117
+
You can measure how long LunAST took to process with the `moonlight.lunast.elapsed` variable
118
+
119
+
### Does this mean patches are dead?
120
+
121
+
No. Patches will continue to serve their purpose and be supported in moonlight, no matter what. LunAST should also work with patches, but patches may conflict or not match.
122
+
123
+
[astring](https://github.com/davidbonnet/astring) may need to be forked in the future to output code without whitespace, in the event patches fail to match on AST-patched code.
124
+
125
+
### This API surface seems kind of bad
126
+
127
+
This is still in heavy development and all suggestions on how to improve it are welcome. :3
128
+
129
+
### Can I help?
130
+
131
+
Discussion takes place in the [moonlight Discord server](https://discord.gg/FdZBTFCP6F) and its `#lunast-devel` channel.
+1
-1
packages/lunast/TODO.md
+1
-1
packages/lunast/TODO.md
···
3
3
- [ ] Experiment more! We need to know what's bad with this
4
4
- [ ] Write utility functions for imports, exports, etc.
5
5
- [ ] Imports
6
-
- [ ] Exports
6
+
- [x] Exports
7
7
- [ ] Constant bindings for an object
8
8
- [ ] Map Z/ZP to default
9
9
- [x] Steal Webpack require and use it in our LunAST instance
+12
-5
packages/lunast/src/index.ts
+12
-5
packages/lunast/src/index.ts
···
17
17
private processors: Processor[];
18
18
private defaultRequire?: (id: string) => any;
19
19
20
+
elapsed: number;
21
+
20
22
constructor() {
21
23
this.modules = {};
22
24
this.types = {};
···
25
27
this.typeCache = {};
26
28
this.fieldCache = {};
27
29
this.processors = getProcessors();
30
+
31
+
this.elapsed = 0;
28
32
}
29
33
30
34
public static getVersion() {
···
34
38
}
35
39
36
40
public parseScript(id: string, code: string): string | null {
41
+
const start = performance.now();
42
+
37
43
const available = [...this.processors]
38
44
.sort((a, b) => (a.priority ?? 0) - (b.priority ?? 0))
39
45
.filter((x) =>
···
68
74
}
69
75
}
70
76
71
-
if (dirty) {
72
-
return generate(module);
73
-
} else {
74
-
return null;
75
-
}
77
+
const str = dirty ? generate(module) : null;
78
+
79
+
const end = performance.now();
80
+
this.elapsed += end - start;
81
+
82
+
return str;
76
83
}
77
84
78
85
public getType(name: string) {
+8
packages/lunast/src/modules/test.ts
+8
packages/lunast/src/modules/test.ts
+2
-2
packages/lunast/src/remap.ts
+2
-2
packages/lunast/src/remap.ts
···
3
3
4
4
export type Processor = {
5
5
name: string;
6
-
find?: string | RegExp;
6
+
find?: string | RegExp; // TODO: allow multiple finds
7
7
priority?: number;
8
-
dependencies?: string[];
8
+
dependencies?: string[]; // FIXME: this can skip modules
9
9
process: (state: ProcessorState) => boolean;
10
10
};
11
11
export type ProcessorState = {