A core problem in the development and maintenance of crowd-sourced filter
lists is that their maintainers cannot confidently predict whether (and where)
a new filter list rule will break websites. This is a result of enormity of the
Web, which prevents filter list authors from broadly understanding the impact
of a new blocking rule before they ship it to millions of users. The inability
of filter list authors to evaluate the Web compatibility impact of a new rule
before shipping it severely reduces the benefits of filter-list-based content
blocking: filter lists are both overly-conservative (i.e. rules are tailored
narrowly to reduce the risk of breaking things) and error-prone (i.e. blocking
tools still break large numbers of sites). To scale to the size and scope of
the Web, filter list authors need an automated system to detect when a new
filter rule breaks websites, before that breakage has a chance to make it to
end users.
In this work, we design and implement the first automated system for
predicting when a filter list rule breaks a website. We build a classifier,
trained on a dataset generated by a combination of compatibility data from the
EasyList project and novel browser instrumentation, and find it is accurate to
practical levels (AUC 0.88). Our open source system requires no human
interaction when assessing the compatibility risk of a proposed privacy
intervention. We also present the 40 page behaviors that most predict breakage
in observed websites.