-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
354 lines (354 loc) · 21.4 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<script src="https://www.w3.org/Tools/respec/respec-w3c" async class="remove"></script>
<link rel="icon" href="data:image/svg+xml,<svg xmlns=%22http://www.w3.org/2000/svg%22 viewBox=%220 0 100 100%22><text y=%220.9em%22 font-size=%22105%22>🪽</text></svg>">
<title>Web Infrastructure Search Endowment (WISE)</title>
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:site" content="@robinberjon">
<meta name="twitter:creator" content="@robinberjon">
<meta name="twitter:title" property="og:title" content="Web Infrastructure Search Endowment (WISE)">
<meta name="twitter:description" property="og:description" content="Standardizing browser & search public infrastructure.">
<meta name="twitter:image" property="og:image" content="https://darobin.github.io/wise/wise-wing.jpg">
<meta name="twitter:image:alt" content="A cute puppy face drawing">
<meta name="twitter:url" property="og:url" content="https://darobin.github.io/wise/">
<meta property="og:locale" content="en">
<style>
body {
background: none !important;
}
</style>
<script class="remove">
var respecConfig = {
specStatus: 'unofficial',
postProcess: [(config, doc) => {
const time = doc.querySelector('#w3c-state time');
const h2 = doc.querySelector('#w3c-state');
h2.innerHTML = 'Proposal ';
h2.appendChild(time);
const dl = doc.querySelector('details > dl');
dl.firstElementChild?.remove();
dl.firstElementChild?.remove();
}],
editors: [{
name: 'Robin Berjon',
url: 'https://berjon.com/',
}],
github: {
repoURL: 'https://github.com/darobin/wise',
branch: 'main',
},
edDraftURI: 'https://darobin.github.io/wise/',
shortName: 'wise',
localBiblio: {
'26-Billion-Default': {
title: 'Google paid a whopping $26.3 billion in 2021 to be the default search engine everywhere',
href: 'https://www.theverge.com/2023/10/27/23934961/google-antitrust-trial-defaults-search-deal-26-3-billion',
authors: ['David Pierce'],
},
AMP: {
title: 'Accelerated Mobile Pages',
href: 'https://amp.dev/',
authors: ['Google'],
status: 'Proprietary Format',
},
'Apple-36': {
title: 'Apple Gets 36% of Google Revenue in Search Deal, Expert Says',
href: 'https://www.bloomberg.com/news/articles/2023-11-13/apple-gets-36-of-google-revenue-from-search-deal-witness-says',
authors: ['Leah Nylen'],
},
'Bad-Search': {
title: 'How bad are search results?',
href: 'https://danluu.com/seo-spam/',
authors: ['Dan Luu'],
},
'Fiduciary-UA': {
title: 'The Fiduciary Duties of User Agents',
href: 'https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3827421',
authors: ['Robin Berjon'],
},
'Internet-Users': {
title: 'Individuals using the Internet (% of population)',
href: 'https://data.worldbank.org/indicator/it.net.user.zs?end=2022&start=1960&view=chart',
authors: ['International Telecommunication Union ( ITU ) World Telecommunication/ICT Indicators Database (via The World Bank)'],
},
'Mozilla-Revenue': {
title: 'Mozilla expects to generate more than $500M in revenue this year',
href: 'https://techcrunch.com/2021/12/13/mozilla-expects-to-generate-more-than-500m-in-revenue-this-year/',
authors: ['Frederic Lardinois'],
},
'Perfect-Webpage': {
title: 'The Perfect Webpage',
href: 'https://www.theverge.com/c/23998379/google-search-seo-algorithm-webpage-optimization',
authors: ['Mia Sato'],
},
'Where-Browsers': {
title: 'Where Browsers Come From',
href: 'https://bkardell.com/blog/WhereBrowsersComeFrom.html',
authors: ['Brian Kardell'],
},
},
};
</script>
</head>
<body>
<section id="abstract">
<p>
Browsers, and the browser engines that power them, provide critical public infrastructure to over five
billion people. To cover the high cost of maintaining such complex feats of engineering, browser
vendors and search engine providers have improvised a system in which money is levied from search
revenue and distributed to browsers. Over the years, this ad hoc system has succeeded in providing some
funding to browsers but it suffers from a muddled lack of transparency and has known detrimental effects
on funding attribution, concentration, search quality, privacy, misinformation, publisher revenue, and
opinion pluralism. This specification captures the web community's experience with the current system so
as to refine requirements and formalizes a browser/search levy that improves on the current one and
addresses its shortcomings.
</p>
</section>
<section id="sotd"></section>
<section>
<h2>Introduction</h2>
<p>
As of 2022, 63% of the world population, or roughly 5 billion people, used the Internet ([[?Internet-Users]]).
While some of these may not use a <dfn data-lt="browser">browser</dfn> (the application used to navigate the
web) regularly, all use a <dfn>browser engine</dfn> (the software component that both browsers and other apps
use to render web content) regularly as those are widely used in applications in addition to forming the
core of [=browsers=].
</p>
<p>
[=Browsers=] and [=browser engines=] are provided to people free of charge, as public goods. Their continued
provision is critical since without them the web collapses. They are also complex: for instance, in 2022
Chromium (the [=browser engine=] powering Google Chrome and other [=browsers=]) ran to about 35 million lines
of code. The total annual cost of maintaining the three primary browser engines is estimated to be around
$2 billion USD ([[?Where-Browsers]]). While this cost is significant, it is only a small fraction of the
direct value (monetary or otherwise) produced by the web.
</p>
<p>
In order to assemble the funds required to operate, almost all browser vendors rely on variations on the
same strategy: they exercise an ad hoc levy on search engine revenue. Variations on this levy include
selling the search engine [=default=] (the initial and typical approach), [=royalties=] on search volume,
or [=intra-company transfers=] (when the search engine and browser belong to the same company). Taken
together, these strategies form the [=AHLD=] system, which is described further in the next section.
</p>
<p>
Despite its critical importance to web infrastructure, the [=AHLD=] system is opaque and poorly documented,
and suffers from a number of undesirable shortcomings that have large-scale detrimental effects on the web.
By formalizing this levy and deliberately architecting it to produce improved results, we can
simultaneously put the funding of critical public digital infrastructure on much surer footing and address
the [=AHLD=] system's issues.
</p>
</section>
<section>
<h2>The <em>Ad Hoc Levy on Defaults (AHLD)</em> System</h2>
<p>
The <dfn data-lt="AHLD">Ad Hoc Levy on Defaults</dfn> (AHLD) is a system in which a portion of search engine
revenue is levied and used to pay browsers. (It is also used for operating systems and other agents, but our
focus in this document is browsers.) The levy ensures that browsers, which constitute criticial infrastructure
for the web, are provided free of charge to people. It can take multiple typical forms:
</p>
<ul>
<li>
<dfn data-lt="default">Default placement</dfn>: the browser vendor sells its default search function to a
search engine. Defaults have an outsized impact on search volume as few users change them. This practice
also includes selling a presence (that isn't the default position) on the list of search engine alternatives
that browsers provide.
</li>
<li>
<dfn>Royalties</dfn>: the browser vendor gets a share of the revenue when directing traffic to a given search
engine.
</li>
<li>
<dfn>Intra-company transfers</dfn>: the browser vendor and search engine vendor are the same company, and
revenue from the latter contributes to paying for the former. In effect, this is a horizontally
integrated variant of [=default placements=] that is cheaper and of greater strategic value. (For smaller
search engines, it may also be the only deal they can make.)
</li>
</ul>
<p>
These deals are negotiated bilaterally, and in practice an [=AHLD=] agreement between a browser vendor and
a search engine vendor can marry aspects from several of these forms.
</p>
<p>
[=AHLD=] levies are economically significant and constitute one of the larger commercial exchanges on the
web. For instance, in the year 2021 Google Search spent over $26 billion USD on [=default placement=] deals
([[?26-Billion-Default]]) — which does not include its [=intra-company transfer=] costs — and paid 36% of
its search advertising revenue made in Safari to Apple ([[?Apple-36]]). In 2020, Mozilla made 86% of its
revenue from the Google Search levy ([[?Mozilla-Revenue]]). <span class="note">Note: We have to use relatively
old numbers due to the unfortunate lack of transparency that the [=AHLD=] system suffers from. To the best
of our knowledge, the situation has not significantly evolved since.</span>
</p>
<section>
<h2>Reviewing the AHLD</h2>
<p>
The [=AHLD=] levy was an excellent invention and remains a great idea twenty years on. Taking a system view of
the web, search is indeed one of the logical places at which to apply a levy. Search extracts value from
content and behavior on the web (it would have no value otherwise) and renders it available at a functional
choke point (it is one of the required components of web discovery). In turn, the value of web content is
only possible because browsers provide critical infrastructure services that render that content available and
attractive to people, and for the most part safe. In an idealized view, we can envision a virtuous, regenerative
cycle: the search levy finances core web infrastructure and that core web infrastructure makes the web successful
such that search engines enjoy strong businesses that can be levied from.
</p>
<p>
Unfortunately, the [=AHLD=] levy has failed to evolve as the web grew from a time of relative infancy when only
14% of the world population (fewer than a billion, [[?Internet-Users]]) used the Internet to the essential
part of society that it is today. Over time, serious shortcomings with this ad hoc approach have surfaced
that have not been addressed. It is incumbent on the web community to step up and hammer out an agreement for
a better levy, as well as to commit to maintaining it so as to avoid the accumulation of problems ([[rfc9413]]).
The changes need not be revolutionary since the core principle — a levy on search to pay for web infrastructure —
can readily remain the same. However, as pressure mounts to address the problems caused by an unmaintained, ad hoc
system, if we fail to act we may lose the levy altogether. Should that happen, the web will suffer from the impact
on its infrastructure and we may find it very difficult to maintain a high-quality browser engine, let alone a
diverse set of them.
</p>
<p>
The rest of this section captures the shortcomings of the [=AHLD=] levy.
</p>
<p>
Search is a key architectural component of the web, browsers provide critical infrastructure, but unfortunately
there is <strong>no transparency</strong> into the system. Almost everything that the web community knows about
[=AHLD=] we know thanks to material released via court cases. If we are to take seriously the W3C's goals of
building a web for all humankind, we need to ensure that the web community is able to evaluate how the beating
heart of web infrastructure operates.
</p>
<p>
There is also <strong>no accountability</strong> concerning the bilateral ad hoc deals made within [=AHLD=]. A
search engine may impose additional requirements on browser vendors that might not be in users' best interests
(e.g. that the browser must not implement certain privacy protections) without oversight. A system deployed at
such a scale needs to be trustworthy.
</p>
<p>
It <strong>does not respect the priority of constituencies</strong> ([[?ethical-web-principles]]). Search deals
made via the levy can have terms that forbid the browser from intervening on the search engine results page
(SERP) even when such changes would be beneficial to the user. For instance, a search engine may benefit from
making its ads look like search results or from pushing users to be logged in, sacrificing privacy. A browser
is expected to counteract such practices, but the [=AHLD=] levy prevents that.
</p>
<p>
The system <strong>gives power to search engines over how browsers work</strong>, including over aspects of
browsers that may not seem search-related, for instance privacy features that can be used for advertising
attribution may have to be approved by the search engine before they can ship in a browser. The lack of
public accountability over the royalties part of the system means that there is a strong information
asymmetry between the search engine and the browser in terms of where the royalties come from and what may
affect them, which empowers the search engine to threaten potential revenue loss when the browser makes a
change they dislike in ways that cannot be verified. Because of this, <strong>the search engine is both judge
and party</strong> in the relationship, and in a position to exert undue influence over browsers.
</p>
<p>
Browsers are paid by the [=AHLD=] levy and may be subject to private requirements imposed by search engines, but
from the perspective of the web community <strong>no requirements are placed on browsers</strong> in
exchange for benefitting from the system.
</p>
<p>
The levy system pays for browsers but <strong>it does not pay for browser engines</strong>. [=Browser engines=]
are the more complicated component and ought to be supported directly. As things stand, a [=browser=] can use
an open source [=browser engine=], collect funds from the levy, but not contribute anything back to the
browser engine. This free riding is detrimental to the maintenance of a rich ecosystem of browser engines.
</p>
<p>
Worryingly, <strong>most appropriated funds don't go towards web infrastructure</strong>. While the purpose of
the levy is manifestly to support web infrastructure, the overwhelming majority of funds is directed elsewhere.
To take but one example: based on 2021 numbers, of the $26 billion USD that Google Search paid in levy, $18
billion went to Apple (about 70%). There is scant evidence that Apple, a publicly-traded company, spends in
the vicinity of $18 billion USD per year on web infrastructure. The levy therefore suffers from very low
efficiency, and in turn this causes value produced on the web to be directed outside of the web, which
subsidizes its proprietary competitors.
</p>
<p>
The logic of applying a web infrastructure levy on search is because search can be considered to distill the
value produced by publishers, ecommerce sites, and content creators. Unfortunately, <strong>publishers,
ecommerce sites, and content creators have no say in a system eventually build on their work</strong>.
While indexing content for the purpose of providing links back to the original source is evidently in
everyone's interest, the lack of checks and balances in the system has led to a race to the bottom in
which search engines provide fewer-and-fewer links back to sources ([[?AMP]]) while increasing the number
of non-linking purposes for which they process the indexed content, such as generative AI.
</p>
<p>
Last but not least, the manner in which the [=AHLD=] system is structured means that <strong>it mechanically
increases concentration in the search market</strong>. Few people change their default search engine (particularly
on mobile devices), which means that purchasing a [=default placement=] effectively purchases market share.
In turn, the search engine with the highest market share has greater profits from which to pay a higher
price for further [=default placements=]. This eventually leads to almost every browser defaulting to the
same search engine, and that search engine dominating the web by insuperable margins. In turn, this
artificially-crated and -sustained concentration creates further problems:
</p>
<ul>
<li>
<strong>Decreasing search quality</strong>. It is both much easier and far more valuable for sites to optimize
for a single ranking function that changes occasionally than for many different ones that evolve in different
directions. This makes signals of content quality easier to fake, thereby empowering low-quality sites to
rate higher. The decrease in search quality ([[?Bad-Search]]) and the effect of search monoculture on the web
([[?Perfect-Webpage]]) are well-documented.
</li>
<li>
<strong>Higher misinformation</strong>. As signals of quality lose their value, it becomes increasingly easier
for misinformation to weave its way into search results.
</li>
<li>
<strong>Loss of pluralism</strong>. A search engine defines an editorial policy: like newspapers or TV channels
before it, it determines what is and isn't relevant. The modality differs in that the editorial policy is
applied on demand and automatically, but it is no different from and no less subjective than that set by other
forms of media. Having close to a single editorial policy for the entire web eliminates media pluralism, which
has negative impacts on society and democracy, and leads to privatized censorship.
</li>
<li>
<strong>One company pays for all browsers</strong>. Every major [=browser engine=] and [=browsers=] accounting
for at least 90% of the market are all paid for by one single search engine. Without ascribing malice or asserting
misbehavior, this level of privatized control over critical infrastructure is incompatible with a free and
equal Internet.
</li>
</ul>
<p>
While it is necessary to take stock of the [=AHLD=]'s shortcomings, and while having left it to linger too long in
outdated ad hoc arrangements has caused these shortcomings to fester, these problems, once identified, can be
addressed. The web community has a responsibility to do so quickly and effectively, as well as durably.
</p>
</section>
</section>
<section>
<h2>Requirements for a <em>Web Infrastructure Search Endowment (WISE)</em></h2>
<p>
tk
MAKE THIS JUST REQUIREMENTS
</p>
<!--
- make sure this is a theory of change
- transparency
- publishers represented
- funding structure
- earmarks for engines
- multihoming
- verticals
- fiduciary duties
- auditability of revenue
- respect for the Priority
- (go though issues to make this list)
- driving towards search pluralism
-->
<section>
<h2>Operational Outline of the Web Fund</h2>
<p>
tk
</p>
</section>
</section>
<section class="appendix">
<h2>Notes on Choice Dialogs</h2>
<p>
tk simply show how they fail to meet requirements at a satisfactory level
lots of testimony from Apple and Mozilla (cite transcripts) that they aren't user-friendly
</p>
</section>
<section class="appendix">
<h2>Acknowledgements</h2>
<p>
The following people (in alphabetical order) have provided invaluable input into this draft:
Dietrich Ayala,
Matthew Frehlich,
and
Max Gendler.
</p>
</section>
</body>
</html>