contentious.com

To compensate for our government’s human-unfriendly info systems, some people have developed civic info-filtering backup systems: news organizations, activists, advocacy groups, think tanks, etc.

In my opinion, ordinary Americans have come to rely too heavily on these third parties to function as our “democracy radar.” We’ve largely shifted to their shoulders most responsibility to clue us in when something is brewing in government, tell us how we can exercise influence (if at all), and gauge the results of civic and government action.

Taken together, these backup systems generally have worked well enough — but they also have significant (and occasional dangerous) flaws. They’ve got too many blind spots, too many hidden agendas, insufficient transparency, and too little support for timely, effective citizen participation…

In other words, the patchwork network of backup systems often fail to supply enough civic information to precisely those people who are most likely to be involved or affected by civic issues, in ways that engage them and support participation. Also, often the civic info they offer generally reflects the providers’ own agendas, assumptions, habits, and preferences — about which they may or may not be conscious or transparent.

Yes, having these backup civic info systems is certainly better than relying solely on the government’s own information systems — but too often, not by much. And sometimes they can even be much worse.

JUST GIVE ME THE DATA

The Knight Foundation has been supporting some efforts to make civic and public info more user-friendly and direct, like Adrian Holovaty’s Everyblock project. This is another third-party civic info “backup system” that aims to provide a more direct experience of civic info. They try (and mostly succeed) to improve upon government communications by enhancing relevance and usability. Everyblock empowers users to search and filter civic info as they choose (at least within a geographic context).

…But there’s a big catch to offering this valuable service: Everyblock must cope with the fact that usually getting raw civic info from government and public sources is a huge pain. It requiring considerable tweaking and maintenance to constantly adapt their “screen scraping” processes.

Screen scraping is a painstaking, cumbersome programming technique. A screen scraper program extracts data from the final display output of another program (what gets shown in, say, your web browser). According to Wikipedia: “The key element that distinguishes screen scraping from regular parsing is that the output being scraped was intended for final display to a human user, rather than as input to another program, and is therefore usually neither documented nor structured for convenient parsing.” That means the whole process is inherently pitfall-prone and inefficient.

Blogger and author Jon Udell nailed the underlying problem of data friction inherent in situations where civic media are forced to resort to screen scraping to obtain public information:

“Data friction can be intentional or not. When it’s intentional, you might have to file a FOIA request to get it. But in a lot of cases, it’s unintentional. The data is public, and intended to be widely seen and used, but isn’t readily reusable.

“…Now it’s time to grease the wheels. Here’s one way that can happen. An enlightened city government can decide to publish [its] data in a reusable way. I’ve written extensively about Washington DC’s groundbreaking DCStat program which does exactly that. I can’t wait to see what happens when EveryBlock goes to Washington.

“But city governments shouldn’t have to go out of their way to provide web-facing data services and feeds. Databases should natively support them. That’s the idea behind Astoria (ADO.NET Services), which is discussed in this interview with Pablo Castro. If the NYC Department of Health had that kind of access layer sitting on top of its [restaurant inspection] database, it wouldn’t put EveryBlock’s screen-scraper out of a job — it would just make that [person's] job a whole lot more interesting and effective.”

This all leads back to why I like what the Knight Commission on the Information Needs of Communities in a Democracy is doing: They’re flipping the focus around, to put people’s needs first.

They’re doing this by starting from the question “What kind of information do communities need?” — rather than simply settling for “How can we tweak the badly designed, human-unfriendly entrenched patchwork system of civic information so that it becomes at least slightly less painful or more useful?”

…Well, they’re doing that to a point, anyway. The crucial limitation I see in their approach lies in how the Knight Commission has chosen to define “community.”

(COMING THURSDAY: Part 3, Beyond Geography…)