davidgasquez.com
📚 Personal bits of knowledge
🎨 Restructure Impact Evaluators for improved readability and organization
- Move "Start local and iterate" section higher with expanded context
- Add bold headers to key concepts for better scanning
- Consolidate related content under appropriate sections
- Remove duplicate entries and improve logical flow
+18
-21
Impact Evaluators.md
+18
-21
Impact Evaluators.md
···
16
16
- Do more to make different aspects of alignment legible, while not centralizing in one single "watcher", we can make the concept much more effective, and fair and inclusive in the way that the Ethereum ecosystem strives to be.
17
17
- Impact Evaluators need to be (permissionless) forkable.
18
18
- Anyone should be able to [fork the evaluation system with their own criteria](https://vitalik.eth.limo/general/2024/09/28/alignment.html), preventing capture and enabling experimentation.
19
-
- Start local and a small community and grow from there.
19
+
- **Start local and iterate**. Begin with small communities defining their own [[Metrics]] and evaluation criteria. Use rapid [[Feedback Loops]] to learn what works before scaling up.
20
+
- Each community understands its context better than outsiders ([seeing like a state blinds you to local realities](https://slatestarcodex.com/2017/03/16/book-review-seeing-like-a-state/))
21
+
- Local experiments surface patterns for higher-level systems
22
+
- Small groups enable iterated games that reward trust and penalize defection
23
+
- Reduced size reduce friction
20
24
- Impact evaluation should be done by the community at the local level. They should define their own metrics and evaluation criteria.
21
25
- IEs should start small (community) and simple. Iterate as fast as possible with a learning feedback loop (there isn't a community one in deepfunding)!
22
26
- Figure out system structures and incentives and use as an examples for the level above.
···
27
31
- [IEs are the scientific method in disguise like AI evals](https://eugeneyan.com/writing/eval-process/). You need automated IEs, which is basically science applied to building better systems. You also need human oversight.
28
32
- For areas with continuous output (e.g: minting for "better path finding algorightms"), follow Bittensor model.
29
33
- IEs are like nuclear power: extremely powerful if used correctly, but so very easy to get wrong, and when things go wrong the whole thing blows up in your face.
30
-
- **Start local and iterate**. Begin with small communities defining their own [[Metrics]] and evaluation criteria. Use rapid [[Feedback Loops]] to learn what works before scaling up.
31
-
- Each community understands its context better than outsiders ([seeing like a state blinds you to local realities](https://slatestarcodex.com/2017/03/16/book-review-seeing-like-a-state/))
32
-
- Local experiments surface patterns for higher-level systems
33
-
- Small groups enable iterated games that reward trust and penalize defection
34
-
- Reduced size reduce friction
35
34
- **Build anti-Goodhart resilience**. Any metric used for decisions [becomes subject to gaming pressures](https://en.wikipedia.org/wiki/Campbell%27s_law). Design for evolution:
36
35
- Run multiple evaluation algorithms in parallel and let humans choose
37
36
- Use exploration/exploitation trade-offs (like multi-armed bandits) to test new metrics
···
44
43
- Multiple communities to share measurement infrastructure
45
44
- Different evaluation methods to operate on the same data
46
45
- Evolution through recombination rather than redesign
47
-
- We might be in an "Arrow's Impossibility Theorem" situation where there is no way to design a mechanism that is fair, efficient and incentive compatible.
46
+
- **Embrace plurality over perfection**. [No single mechanism can satisfy all desirable properties](https://en.wikipedia.org/wiki/Arrow%27s_impossibility_theorem) (efficiency, fairness, incentive compatibility, budget balance). Different contexts need different trade-offs.
48
47
- There is no "end of history" because whenever you fix an evaluation, some group has an incentive to abuse or break it again and feast on the wreckage.
49
48
- This is the formal impossibility theorem that no mechanism can simultaneously achieve four desirable criteria:
50
49
- Pareto Efficiency: The outcome achieved by the mechanism maximizes the overall welfare or some other desirable objective function.
51
50
- Incentive Compatibility: Designing mechanisms so that participants are motivated to act truthfully, without gaining by misrepresenting their preferences.
52
51
- Individual Rationality: Ensuring that every participant has a non-negative utility (or at least no worse off) by participating in the mechanism.
53
52
- Budget Balance: The mechanism generates sufficient revenue to cover its costs or payouts, without running a net deficit.
54
-
- [Current markets can optimize for the wrong things](https://meaningalignment.substack.com/p/market-intermediaries-a-post-agi) - clicks, engagement, and profit - rather than human wellbeing and flourishing. This misalignment will become catastrophic as AI systems become more powerful (arms races, propaganda, ...)
55
-
- Three common responses to market failures, all of which centralize power without fixing the underlying incentive problems: coordination, redistribution, and regulation.
56
-
- The wrong outcomes are being monitored and contracted against. Is hard to optimize for "human-flourishing-related outcomes".
57
-
- Four interconnected issues:
58
-
- Incomplete contracts - It's too expensive to measure what really matters (human flourishing), so we contract on proxies (hours worked, subscriptions)
59
-
- Power asymmetries - Large suppliers face millions of individual consumers with take-it-or-leave-it contracts
60
-
- Externalities - Individual flourishing depends on community wellbeing, but contracts remain individualized
61
-
- Information asymmetries - Suppliers control the metrics and optimize for growth rather than user outcomes
53
+
62
54
- When collecting data, [pairwise comparisons and rankings are more reliable than absolute scoring](https://anishathalye.com/designing-a-better-judging-system/). Humans excel at relative judgments, but struggle with absolute judgments.
63
-
- Legibility is key. E.g: How much did X contribute to Ethereum? The goal is to transform alignment from an exclusive social game into a merit-based system with clear, measurable criteria.
55
+
- **Legible Impact Attribution**. Make contributions and their value visible. [Transform vague notions of "alignment" into measurable criteria](https://vitalik.eth.limo/general/2024/09/28/alignment.html) that projects can compete on.
64
56
- Support organizations like L2beat to track project alignment
65
57
- Let projects compete on measurable criteria rather than connections
66
58
- Enable neutral evaluation by EF and others
···
72
64
- **Incomplete contracts problem**. [It's expensive to measure what really matters](https://meaningalignment.substack.com/p/market-intermediaries-a-post-agi), so we optimize proxies that drift from true goals.
73
65
- Current markets optimize clicks and engagement over human flourishing
74
66
- The more powerful the optimization, the more dangerous the misalignment
67
+
- Four interconnected issues:
68
+
- Incomplete contracts - It's too expensive to measure what really matters (human flourishing), so we contract on proxies (hours worked, subscriptions)
69
+
- Power asymmetries - Large suppliers face millions of individual consumers with take-it-or-leave-it contracts
70
+
- Externalities - Individual flourishing depends on community wellbeing, but contracts remain individualized
71
+
- Information asymmetries - Suppliers control the metrics and optimize for growth rather than user outcomes
75
72
- **Information elicitation without verification**. Getting truthful data from subjective evaluation when you can't verify it requires clever [[Mechanism Design]]:
76
73
- [Peer prediction mechanisms](https://jonathanwarden.com/information-elicitation-mechanisms/) that reward agreement with hidden samples
77
74
- [Bayesian Truth Serum](https://www.science.org/doi/10.1126/science.1102081) that uses both answers and predictions
···
91
88
- IE's helps a community with more data and information to make better decisions.
92
89
- Open Data Platforms for the community to gather better data and make better decisions.
93
90
- Can open data be rewarded with an IE? What does a block reward mean there?
94
-
- Legible Impact Attribution. Make contributions and their value visible. [Transform vague notions of "alignment" into measurable criteria](https://vitalik.eth.limo/general/2024/09/28/alignment.html) that projects can compete on.
95
-
- Seeing like a State blinds you to the realities that are complex. Need a way to evolve the metric to be anti-Goodhart's.
91
+
- **Legible Impact Attribution**. Make contributions and their value visible. [Transform vague notions of "alignment" into measurable criteria](https://vitalik.eth.limo/general/2024/09/28/alignment.html) that projects can compete on.
92
+
- Seeing like a State blinds you to the realities that are complex. Need a way to evolve the metric to be anti-Goodhart's.
96
93
- Not even anti-goodharts. Research says the best thing to do is to give all money to vaccine distribution, ...
97
94
- **Embrace plurality over perfection**. [No single mechanism can satisfy all desirable properties](https://en.wikipedia.org/wiki/Arrow%27s_impossibility_theorem) (efficiency, fairness, incentive compatibility, budget balance). Different contexts need different trade-offs.
98
95
- **Make evaluation infrastructure permissionless**. Just as anyone can fork code, anyone should be able to fork evaluation criteria. This prevents capture and enables innovation.
···
100
97
- **Layer human judgment on algorithmic engines**. The ["engine and steering wheel" pattern](https://vitalik.eth.limo/general/2025/02/28/aihumans.html) - let algorithms handle scale while humans set direction and audit results.
101
98
- The easier to verify the solution is (e.g: verify a program passes the test vs verify the experiment replicates), the better and faster the IE can be.
102
99
- If the domain of the IE is sortable and differentiable, it's easy as it can be seen as pure optimization and doesn't require humans subjective evaluation.
103
-
- Verify the evaluation is actually better than the baseline.
100
+
- **Verify the evaluation is actually better than the baseline**.
104
101
- Run multiple "aggregations" algorithms and have humans blindly select which one they prefer (blind test).
105
102
- The meta-layer can help compose and evaluate mechanisms. How do we know mechanism B is better than A? Or even better than A + B, how do we evolve things?
106
103
- Reinforcement Learning?
···
108
105
- Is the evaluation/reward better than a centralized/simpler alternative?
109
106
- E.g: on tabular clinical prediction datasets, standard logistic regression was found to be on par with deep recurrent models
110
107
- [IEs need to show how the solution is produced by the interactions of people each of whom possesses only partial knowledge](https://news.ycombinator.com/item?id=44232461).
111
-
- IEs are optimization processes with tend to exploit (more impact, more reward). This ends up with a monopoly (100% exploit). You probably want to always have some exploration.
108
+
- **Exploration vs Exploitation**. IEs are optimization processes with tend to exploit (more impact, more reward). This ends up with a monopoly (100% exploit). You probably want to always have some exploration.
112
109
- Do IEs need some explore/exploit thing? E.g. Use multi-armed bandit algorithms to adaptively choose between evaluation mechanisms based on historical performance and context.
113
110
- Use maximal lotteries to enforce the exploration
114
111
- Having discrete rounds simplify the process. Like a batch pipeline.
115
112
- The more humans gets involved, the messier (papers, ... academia). You cannot get away from humans in most problems.
116
113
- [Campbell's Law](https://en.wikipedia.org/wiki/Campbell%27s_law). The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.
117
114
- [The McNamara Fallacy](https://en.wikipedia.org/wiki/McNamara_fallacy). Never choose metrics on the basis of what is easily measurable over what is meaningful. Data is inherently objectifying and naturally reduces complex conceptions and process into coarse representations. There’s a certain fetish for data that can be quantified.
118
-
- IEs should define also a Data Structure for each layer so they can compose (graph, weight vector). That is the API.
115
+
- **Composable Data Structures**. IEs should define also a Data Structure for each layer so they can compose (graph, weight vector). That is the API.
119
116
- E.g: Deepfunding problem data structure is a graph. Weights are a vector/dict, ...
120
117
- IEs will have to do some sort of "error analysis". [Is the most important activity in LLM evals](https://hamel.dev/blog/posts/evals-faq/#q-why-is-error-analysis-so-important-in-llm-evals-and-how-is-it-performed). Error analysis helps you decide what evals to write in the first place. It allows you to identify failure modes unique to your application and data.
121
118
- Film festivals are "local" IEs each one serving different values/communities.