Being a Citizen Shouldn’t Be So Hard! Part 2: Beyond Government

NOTE: This is part 2 of a multipart series. See the series intro. More to come over the next few days.

This series is a work in process. I’m counting on readers and others to help me sharpen this discussion so I can present it more formally for the Knight Commission to consider.

So please comment below or e-mail me to share your thoughts and questions. Thanks!

To compensate for our government’s human-unfriendly info systems, some people have developed civic info-filtering backup systems: news organizations, activists, advocacy groups, think tanks, etc.

In my opinion, ordinary Americans have come to rely too heavily on these third parties to function as our “democracy radar.” We’ve largely shifted to their shoulders most responsibility to clue us in when something is brewing in government, tell us how we can exercise influence (if at all), and gauge the results of civic and government action.

Taken together, these backup systems generally have worked well enough — but they also have significant (and occasional dangerous) flaws. They’ve got too many blind spots, too many hidden agendas, insufficient transparency, and too little support for timely, effective citizen participation…

In other words, the patchwork network of backup systems often fail to supply enough civic information to precisely those people who are most likely to be involved or affected by civic issues, in ways that engage them and support participation. Also, often the civic info they offer generally reflects the providers’ own agendas, assumptions, habits, and preferences — about which they may or may not be conscious or transparent.

Yes, having these backup civic info systems is certainly better than relying solely on the government’s own information systems — but too often, not by much. And sometimes they can even be much worse.


The Knight Foundation has been supporting some efforts to make civic and public info more user-friendly and direct, like Adrian Holovaty’s Everyblock project. This is another third-party civic info “backup system” that aims to provide a more direct experience of civic info. They try (and mostly succeed) to improve upon government communications by enhancing relevance and usability. Everyblock empowers users to search and filter civic info as they choose (at least within a geographic context).

…But there’s a big catch to offering this valuable service: Everyblock must cope with the fact that usually getting raw civic info from government and public sources is a huge pain. It requiring considerable tweaking and maintenance to constantly adapt their “screen scraping” processes.

Screen scraping is a painstaking, cumbersome programming technique. A screen scraper program extracts data from the final display output of another program (what gets shown in, say, your web browser). According to Wikipedia: “The key element that distinguishes screen scraping from regular parsing is that the output being scraped was intended for final display to a human user, rather than as input to another program, and is therefore usually neither documented nor structured for convenient parsing.” That means the whole process is inherently pitfall-prone and inefficient.

Blogger and author Jon Udell nailed the underlying problem of data friction inherent in situations where civic media are forced to resort to screen scraping to obtain public information:

“Data friction can be intentional or not. When it’s intentional, you might have to file a FOIA request to get it. But in a lot of cases, it’s unintentional. The data is public, and intended to be widely seen and used, but isn’t readily reusable.

“…Now it’s time to grease the wheels. Here’s one way that can happen. An enlightened city government can decide to publish [its] data in a reusable way. I’ve written extensively about Washington DC’s groundbreaking DCStat program which does exactly that. I can’t wait to see what happens when EveryBlock goes to Washington.

“But city governments shouldn’t have to go out of their way to provide web-facing data services and feeds. Databases should natively support them. That’s the idea behind Astoria (ADO.NET Services), which is discussed in this interview with Pablo Castro. If the NYC Department of Health had that kind of access layer sitting on top of its [restaurant inspection] database, it wouldn’t put EveryBlock’s screen-scraper out of a job — it would just make that [person’s] job a whole lot more interesting and effective.”

This all leads back to why I like what the Knight Commission on the Information Needs of Communities in a Democracy is doing: They’re flipping the focus around, to put people’s needs first.

They’re doing this by starting from the question “What kind of information do communities need?” — rather than simply settling for “How can we tweak the badly designed, human-unfriendly entrenched patchwork system of civic information so that it becomes at least slightly less painful or more useful?”

…Well, they’re doing that to a point, anyway. The crucial limitation I see in their approach lies in how the Knight Commission has chosen to define “community.”

(COMING THURSDAY: Part 3, Beyond Geography…)

7 thoughts on Being a Citizen Shouldn’t Be So Hard! Part 2: Beyond Government

  1. Pingback:   Being a Citizen Shouldn’t Be So Hard! Part 1: Human Nature  —

  2. In my own experience, few people think they suffer from lack of information. Only in very specific instances (unexpectedly high bills from sewer hookups, for example) do most people actively seek more information than they are already getting.

    If we believe the problem is citizens not getting the right information or it not being delivered in user-friendly ways, we’ll have to make a convincing argument about why and how more information is going to make a difference.

    One model of democracy assumes people are interested citizens who seek information to develop rational opinions about public affairs. Other models show that people form emotional and personal interests in a subject and then seek information — not the other way around. As Schudson describes it, they use the media for monitorial purposes — turning to the news when something of high personal interest happens but otherwise paying only enough attention to know if the world is blowing up or not. This sounds a little like the “democracy radar” you describe.

    I wonder, given the evolution of networks and groups, whether speaking at the level of “communities” and “people” is specific enough to be useful anymore. It sounds a little like the mass media model, assuming a public “out there” that we only imagine but doesn’t really exist. If publics are becoming more fluid and responsive to specific issues, perhaps we need more journalism that recognizes and supports this development.

  3. …perhaps we need more journalism that recognizes and supports this development…

    Thanks Donica. That’s the point I tried to make in Part I. And while Amy’s comment remains true that “We’ve largely shifted to (third parties/journalists’) shoulders most responsibility to clue us in when something is brewing in government,” it’s not for no reason that’s happened.

    Maybe as a journo I’m just too inside the forest to see anything but the trees. There can be other, more direct ways to connect users to information, but with increased transparency in journalism we can offer more of those connections without inserting ourselves as filters.

    …They’re doing this by starting from the question “What kind of information do communities need?”

    Sounds right.

  4. Amy, I really love this discussion, but have to offer one amendment. You state that the Knight Commission is starting with the question, “What kind of information do communities need?” This is not quite right. We are starting with the question, “What are the information needs of communities in a democracy?” The difference is that the Commission is tending towards the view that “information needs” include not only information per se, but also structures, processes, and enabling conditions through which people can turn information into community empowerment. To elaborate that view, we need, of course, to understand better what constitutes the information infrastructure of geographically defined local communities. Within such communities, all of us undoubtedly do rely, inevitably, on a host of information intermediaries to clue us in to the overwhelming majority of what we regard as our daily knowledge acquisition. The issue, as I see it, is how to maximize the likelihood that our networks of information intermediaries, formal and informal, will be as inclusive, relevant, and trustworthy as possible.

    Best, Peter

  5. Your phrase “too little support for timely, effective citizen participation” says it all. When I conducted a national survey of political researchers in 2001 on the topic of community-based research, their conclusion was that 1. almost no one does research, 2. useful research is rarely found on the Internet or in the media, 3. individuals who do research are usually not activists, and 4. independent research applicable to effective citizen participation is practically never supported by the communities that could use it.

    In my subsequent report, Research as Organizing Tool, I elaborate on these findings for the benefit of those who often feel adrift in a sea of useless information and propaganda. In my essay, The Public Health Model, I explain that the reason useful research is held in such low regard is largely due to the ineffective models of engagement the politically illiterate bring to civic affairs.

  6. I just found this series…great stuff. Did you get anywhere trying to get the Knight Foundation to think about addressing data friction? I didn’t:

    It’s kind of a no-brainer: journalists with big, heavily trafficked news/community sites stand to get “free content” via increasing “gov2.0” data dissemination–pure primary source material for newspapers to analyse openly with/for their audiences, if their web teams are up to it.

    Just up on (John Geraci of started this super project) …

    DIYcity Challenge #7: Open Data!

    “Lots of city agencies all over the world have data online that is accessible to humans in readable format, yet isn’t accessible to other computers and programs via an API. Some agencies don’t have the means to turn their data into an API, others don’t have the inclination to do so.

    Can we help these agencies to open their data?

    DIYcity Challenge #7: build a site scraper for the website of a city agency in your city that scrapes data, dumps it into a database, and offers that to everyone in API format.

    Do not violate any copyrights for this challenge – please only scrape publicly accessible government data, not data from 3rd party sites.

    DIYcity can help host any scraping bots, databases and APIs that come out of this challenge. Or just point us to a dataset you’ve scraped and we’ll make a list in the wiki.”

Leave a Reply

Your email address will not be published. Required fields are marked *