One Week | One Tool: Interim Report

As promised on Twitter, I’m sharing the report (with a few minor copyedits and corrections) I submitted this week to NEH on One Week | One Tool. This is an interim report, which means there are plenty of questions that remain unanswered. The grant continues for another year, during which time the One Week | One Tool crew will be working to support and extend their work together. I’m sure I’ll be blogging about their progress as we go, but at the very least I’ll post again with the final report next summer.

Executive Summary

One Week | One Tool represents a groundbreaking experiment in digital humanities training and digital humanities tool building. Culminating with the institute held July 25 – July 31, 2010 at the Center for History and New Media (CHNM) at George Mason University and the institute participants’ successful construction and release, in seven days, of a completely new open source software product, One Week | One Tool has made significant strides in proving its central claims: 1) that learning by doing is an important and effective part of digital humanities training; 2) that the traditional NEH summer institute can be adapted to accommodate this kind of practical digital humanities pedagogy; and 3) that digital humanities tools can be built more quickly and affordably than common practice would suggest. The second year of the project will seek to reinforce these claims and continue to advance participant learning through ongoing engagement and further development of the tool that was built during their week at CHNM—Anthologize.

Recruitment

The first task of the One Week | One Tool team was to publicize the institute. After dates were chosen, we developed a simple website in the style of an Old West poster to call for proposals. The call for proposals was issued in early February and closed in mid-March. To publicize the call, we leveraged CHNMs extensive outreach networks among digital humanities bloggers and on Twitter, where the announcement generated significant attention and excitement. Applicants were asked to answer three questions: 1) What skills/experiences/interests you think are most important to building a successful tool? 2) Which of these skills/experiences/interests you will bring to the barn raising? 3) What you think you will get out of attending that will help you in future pursuits? From this call, we received 48 formal applications and approximately another dozen serious expressions of interest.

The selection process was remarkably difficult. At least 35 of the 48 applicants were perfectly qualified to attend. To differentiate among them, we stuck to the criteria we outlined in our original proposal and laid out in the call. Applicants were chosen not on the basis of specific qualifications (e.g. a higher degree or particular skill set), but on evidence of teamwork, patience, flexibility, and resourcefulness (such as a history of picking up a programming language on one’s own). We were also looking to assemble a diverse group who together would possess the entire range of skills necessary to conceive, manage, build, and disseminate a tools project. The final group represented a broad range of scholars, students, librarians, museum professionals, developers, user interaction designers, bloggers, and project managers. They were also relatively diverse in terms of gender (4 women, 8 men) and seniority (ranging from a recently graduated college student to a tenured faculty member). Given the importance of intra-team dynamics to the success of software development projects, we took pains to introduce participants to one another via Twitter in the weeks after their acceptance so that they could begin establishing a sense of camaraderie ahead of their intense week of working together in July.

The Institute

Institute participants arrived on Sunday, July 25, 2010 for a kick-off meeting at George Mason University’s brand new Mason Inn and Conference Center, which served as the participants’ home away from home during One Week | One Tool. Participants’ transportation and lodging were arranged and paid ahead of time, with an additional $1000 stipend provided to each participant as an honorarium and to cover any incidentals. After a round of personal and professional introductions, project director Tom Scheinfeldt provided a brief introduction to CHNM and some of the core values CHNM brings to its open source software development work. Particular emphasis was placed on the value of "use," that the best measure of a tool’s success is its uptake and actual use by its intended audience. The first evening concluded with a brief discussion of participant and organizer expectations and an exhortation to get a good night’s sleep.

Monday was originally designed as the "teaching" phase of One Week | One Tool, during which CHNM staff members would provide participants with some lessons in software development best practices. This was ultimately the least successful portion of the program. Although introductory talks on technical aspects of software development (Jeremy Boggs), outreach mechanisms and community building (Trevor Owens), and the state of the art of digital humanities tools (Dan Cohen) were well received, participants were eager to get down to brass tacks, and we made a strategic decision to cut short the lectures and begin brainstorming tool choices a half day early on Monday afternoon.

Brainstorming continued through Monday evening and into Tuesday morning. By mid-morning on Tuesday, the group had proposed dozens of possible directions for the week’s tool building. By lunchtime, it had distilled those ideas down to six possible candidates. Over lunch, the group asked followers on Twitter to vote on the six choices. Well over 50 outside observers recorded their preferences. Taking this outside feedback into account, the group conducted two rounds of voting of its own, making a final consensus decision by mid-afternoon on Tuesday.

Work then turned to choosing roles and responsibilities. With guidance from CHNM staff, by Tuesday evening, participants had arranged themselves into three teams: Development, User Experience, and Outreach. Work began immediately, with developers making choices about encoding and architecture, user experience team members outlining potential use cases, and outreach staff researching the competitive marketplace and writing website copy.

Wednesday, Thursday, and Friday were each long days of code, design, and writing. The group quickly established a daily routine, which started with an all-project check-in meeting at CHNM’s offices at 10:00 a.m., was punctuated by a return to the Mason Inn at about 6:00 p.m., and ended with yet more work
at the hotel bar, usually until well past closing. Communication, leadership, and even membership among the teams were fluid, with teams spontaneously reconfiguring as particular skills were required for particular tasks. Moreover, throughout this "doing" phase, Tom, Dan, Jeremy, and Trevor encouraged additional CHNM staff—including Sharon Leon, Sheila Brennan, Ammon Shepherd, and John Flatness—to be available to help participants solve problems and make tough decisions. By Saturday morning, the One Week | One Tool team had designed and built the software; established an open code repository, a ticketing and bug tracking system, and a set of open source community development channels; and mounted an outreach and marketing campaign that consisted of an original name, logo, and website, as well as promotional bookmarks and stickers, a CafePress storefront, a formal press release, a media contact sheet, a set of public use cases, an FAQ page, and other end-user support channels.

During the course of the week, several members of the team "live blogged" and Tweeted (using the hashtag #oneweek) about this process, but they did so without giving away the final choice of tool, which served to create a buzz about the launch among potential users in the digital humanities community. Between July 25th and August 1st, hundreds of tweets were tagged #oneweek, mostly by outside observers, a measure of the extent to which One Week | One Tool captured the imagination of the digital humanities community. The tool the group produced, Anthologize, was launched on Tuesday, August 3, 2010 with a live video episode of CHNM’s popular Digital Campus podcast. More than 125 people took time out of their work day to tune in to the live announcement. In the days following the announcement, One Week | One Tool and the Anthologize were featured by Atlantic.com, Read Write Web, the Chronicle of Higher Education, and dozens of digital humanities blogs. As one external commentator put it, Anthologize and One Week | One Tool experiment had "set the DH world abuzz."

Anthologize

Anthologize is a free, open-source plugin that transforms WordPress 3.0 into a platform for publishing electronic texts. Anthologize lets users grab posts from their WordPress blog, import feeds from external sites, or create new content directly within Anthologize. Users can then outline, order, and edit their work, crafting it into a single volume for export in several formats, including PDF, ePUB, and TEI. Among Anthologize’s projected users are scholars (to edit anthologies of thematically related blog posts or publish conference proceedings), teachers (to edit student class work into e-portfolios or organize course reading packs), and cultural institutions (to publish exhibit books or donor gift books). Directions for future development include additional export formats, improved importing of external web content, footnote support, and integrating blog comments into the Anthologize editorial workflow.

Learning Outcomes

We are still collecting survey responses and assessing the results, but early on it is clear that One Week | One Tool was a positive learning experience for the participants. Unlike many curriculum-based institutes, the experiences and therefore the learning outcomes at One Week | One Tool varied widely among the participants depending on their team membership and on the skills they brought to the institute. Nevertheless, an initial analysis of participant survey responses reveals some commons themes. Among them:

  • Most participants responded that they learned about collaboration, leadership, and team building. Doug Knox pointed to the "self-taught lessons in group dynamics for a team of pragmatic collaborative autodidacts" he took from One Week | One Tool. As one participant responded, those lessons included "the kind of flexibility, trust, humility, and perseverance that is necessary to take on a relatively big project in a short period of time." In a post titled "Unexpected leadership," Boone Gorges echoed these lessons, writing "leadership doesn’t work without humility and trust." Effie Kapsalis described that same lesson plainly on the Smithsonian 2.0 blog: "Trust. Period."
  • Related to this were the lessons in decision making and calculated risk that most participants said they took away. As one participant wrote, "the experience really made me understand how much fear of acting alone can stall development and how trusting the emergent properties of the system can really lead to fantastic results." Jana Remy wrote in a related vein on her blog, "…one of the most important things we’ve learned through this workshop is the importance of calculated risk. It’s risky to forge forward with a digital project without months of planning, coding, and testing. It’s risky to trust near-strangers to deliver on such a tight deadline…. Our team selected one of the more daring choices available for our project. We were ready to be bold and to dream big. Even if the rest of the world doesn’t see our tool’s potential to be as earthshaking as we do, the process of manifesting our vision for this tool has been a valuable lesson in and of itself, and will certainly carry through to our future endeavors."
  • Several participants responded that they learned about outreach methods, marketing, and dissemination of digital humanities products. As one participant said, "I really found the information on how to manage publicity and outreach extremely useful, and I was completely stunned by the results…."
  • Several participants said they learned particular technologies (e.g. WordPress and Git) and practical lessons in open source software development best practices: "I also think I came to understand a bit more about how agile methodologies really work," remarked one participant.
  • Some participants pointed to their improved understanding of "what constitutes a full end-to-end development cycle." Experienced programmers and project managers, even those who entered the week with hundreds of thousands of lines of code under their belts and long track records of successful digital humanities work, responded that they learned about the differe
    nces between building a tool "from conception to roll-out" as they did at One Week | One Tool and the ad-hoc or task-specific development they do in the normal course of their work.

Every participant that responded to our survey said that One Week | One Tool met or exceeded his or her expectations, that it met its stated goals, that the experience would advance his or her career objectives, and that the model of learning by doing was effective. There were, however, at least two criticisms that were repeated by several participants. First, as mentioned above, most participants would have preferred to cut the Monday lecture portion of the program shorter, something we addressed on the spot and will keep in mind for any possible repeat events. Second, at least two people wanted to see more learning from fellow team members. The fast paced development cycle too often forced participants to keep their noses to the grindstone and opportunities for cross-pollination and peer-to-peer learning were sometimes missed. In any repeat performance of One Week | One Tool we will take seriously one participant’s observation that "…the group itself would have benefited from a more concerted effort to mix up knowledge multilaterally. There was some tension between working in small teams to get work done and making sure we had informal chances to share a sense across groups about the state of progress and the development of our sense of purpose. There were in fact essential unplanned lateral communications in all directions, and the bar was especially important in that. It worked, but there could be room to make it a more reflective process. We should have been teaching our strengths as well as contributing them, and apprenticing in our less-strong areas."

Yet in discussing One Week | One Tool’s learning outcomes, it is also important to note that, perhaps unlike more traditional institutes, the learning appears to have extended well beyond the twelve participants who traveled to Fairfax. This possibly unique aspect of One Week | One Tool results from the fact that: 1) it was an event, and 2) it produced a product for widespread distribution and use. As Tim Carmody posted on Snarkmarket, One Week | One Tool was a "generative web event," an "ongoing communal broadcast." Because it took place over a limited and clearly delineated time period, because it posited a provocative, easy-to-grasp, and unlikely challenge (academics build something in a week?), because it maintained an air of secrecy, and because it was widely publicized, One Week | One Tool was able not only to create "buzz" but to provoke conversation and discussion of serious issues among a broad range digital humanities scholars who could use their common experience of One Week | One Tool as a touchstone for debate. Two examples of this kind of broader discussion are ongoing exchanges about the nature of electronic writing and its relationship to the printed page and about the relative merits of pre- versus post-launch user studies.

Next Steps

Over the next year, the original group of twelve participants will continue to develop the tool from remote locations around the country, forming the core of an open source developer and user community that they will build and support. Indeed, the team has already developed and released a "point release" version 0.4 in the time since they left Fairfax. During the next ten months, the group will fix bugs in the software and add new features. It will continue to disseminate the software through online outreach and in-person conference presentations. As part of the group’s training in digital humanities software sustainability, it will also work with staff at CHNM to write grant proposals for additional funding and implement cost recovery mechanisms where appropriate. In June 2011, the group will travel back to Fairfax for THATCamp 2011 to assess the progress of the past year, gather additional feedback from community members, and make decisions about future development and support of Anthologize.

Conclusion

Overall, we are pleased with the progress of the One Week | One Tool project. The main learning objective of the project was to teach software development and deployment skills to a group of diverse group of digital humanists. As explained to participants on the first evening of the institute, we believe the best measure of the success of a tool is its use by and usefulness to its intended audience. By extension, the best measure of whether a group of people have learned to build successful tools is whether the tool they built is actually useful to and used by its audience. Early indications are good. In less than ten days the Anthologize website has received 232,000 hits, 75,000 page views, and 13,000 unique visitors, and the Anthologize software has been downloaded more than 2,500 times. We have high hopes for the coming year, and we will continue to keep track of these numbers and follow the projects that use Anthologize.

Lessons from One Week | One Tool – Part 3, Serendipity

Over the past few months, several people—including several participants themselves—have asked me how we chose the One Week | One Tool crew. We had about 50 applicants. Nearly all of them were perfectly qualified to attend. This made the selection process exceedingly difficult. I have no doubt we could have ended up with another group of twelve and been equally successful. Indeed, I have often joked that with the applicant pool we got we could easily have done "Three Weeks | Three Tools."

In the end, the uniformly high quality of our applicant pool meant we had to make our choices on largely subjective criteria. Who showed the most enthusiasm? Who showed the most willingness to collaborate? Who seemed open minded? Whose writing style showed clarity of thought and energy? Who seemed like the hardest workers? Who seemed eager to learn? Whose seemed like a quick study?

One important thing to note is that we didn’t pick for particular technical skills. Of course we wanted a good mix: a team full of UI specialists or outreach experts wouldn’t have worked. But we didn’t select for PHP, Ruby, Java, TEI, or any other technology. We assumed that if we picked good people with a range of skills, even if those skills were all over the map, we’d end up with something interesting. I think we have.

As it turns out, we have ended up with something even more interesting because of that diversity of skills. Boone said something very telling at the bar on Thursday night: Anthologize probably couldn’t have been built without One Week | One Tool. A single shop wouldn’t have had the necessary range of skills to do it. I know CHNM couldn’t have done it alone. Sure, we have the PHP and JavaScript chops (although not the intimate knowledge of WordPress Boone brought to the table). But we certainly don’t have deep knowledge of TEI and XSLT the team needed to produce Anthologize’s rich, clean outputs. Nor, I suspect, would we even have hit upon TEI as a solution to that particular problem.

There are lessons here for hiring. Pick the smartest people. The best writers. The hardest workers. Pick people with proven track records of working well in teams. People with lots of energy. People who approach heavy workloads with clear thinking and good humor. People who’ve shown aptitude for picking up new technical skills on their own.

These people may not know what you need them to know on Day 1. But they will work hard to learn it. They’ll also bring a host of additional skills you didn’t think you needed, but will be happy to have once they’re there. Having these additional skills on tap may even let you take your work in directions you couldn’t have otherwise imagined.

#oneweek #buildsomething

Lessons from One Week | One Tool – Part 2, Use

For all the emphasis on the tool itself, the primary aim of One Week | One Tool is not tool building, it’s education. One Week | One Tool is funded by NEH under the the Institutes for Advanced Topics in Digital Humanities (IATDH) program. IATDH grants “support national or regional (multistate) training programs for scholars and advanced graduate students to broaden and extend their knowledge of digital humanities.” Thus training is the criteria by which One Week | One Tool will ultimately be judged.

A key argument of One Week | One Tool is that learning digital humanities consists primarily in doing digital humanities, that digital humanities is a hands-on kind of thing, that to learn tool building you have to do some tool building. At the same time, we recognize that there’s a place for instruction of the hands-off sort. To that end, the first 18 hours or so of One Week | One Tool (essentially from Sunday night until mid-afternoon on Monday) were reserved for presentations by CHNM staff. Jeremy offered a practical introduction to software development best practices and tools. Trevor described the range of outreach strategies we have employed on projects like Zotero, Omeka, and the National History Education Clearinghouse. Dan provided the view from 30,000 feet with thoughts on the state of the art and near future of digital humanities software development. I kicked things off on Sunday with a brief introduction to CHNM and our tool building philosophy. Several strains of thought and practice inform our work at CHNM—public history, cultural history, radical democracy, dot.com atmospherics, and more—but to keep things simple I summed up our tool building philosophy in one word: use.

Here is more or less what I told the crew.

At CHNM we judge our tools by one key metric above all others: use. Successful tools are tools that are used. The databases of Sourceforge and Google Code are littered with interesting, even useful, but unused open source tools. Academic software projects are no exception. Every year NSF, NIH, and now NEH and IMLS award grants for scholarly software development. In recent years, the funding guidelines have stipulated that this software be made freely available under open source licenses. Much of the software produced by these programs is good and useful code. But little of it is actually used.

There are several reasons for this. Many efforts are focused narrowly on the problems of a particular researcher or lab. While the code produced by these researchers proves useful for solving their particular problems, even when released, it hasn’t been designed to be generally applicable to the needs of other researchers in the field. It is, in effect, a one-off tool released as open source. But open source code alone does not constitute an open source project.

Other projects build generalized tools that may be of potential use to other researchers. But few make the necessary investment in outreach and, yes, marketing to make potential users aware of the tool. It is for this reason, among others, that we see so much duplication of effort and functionality in scholarly software projects.

Building a user community is the first prerequisite to building a successful open source software project. The success of software is judged by its use. The universal assessment that iTunes is a hit and Zune is a flop is not based on the quality of the code or even the elegance or potential usefulness of the experience. It’s based on the fact that everybody uses iTunes and nobody uses Zune. This is not to say that software has to have millions of users to be successful. But it is to say that successful software is used by large swath of its potential users. To be sure, the total population of potential users of cultural heritage mapping tools is much smaller than the total population of potential users of digital media playback software. But any open source software project’s goal should be use by as many of its potential users as possible. In any case, we should aim to have our software used by as many cultural heritage institutions and digital humanists as possible.

Moreover, a large and enthusiastic user base is key to a successful open source software project’s continued success. If people use a product, they will invest in that product. They will provide valuable user testing. They will support the project in its efforts to secure financial support. They will help market the product, creating a virtuous circle. Sustainability, even for free software, is grounded in a committed customer base.

Related to building a user community is building an open source developer community. Some number of users will have the inclination, the skills, and the commitment to the project to help on the level of code. This percentage will be very small, of course, less than one percent, which is another reason to build a large user base. But this small group of code contributors and volunteer developers forms the core of most successful open source projects. They find and fix bugs. They provide end user support. They write documentation. They add new features and functionality. They provide vision and critical assessment. They constitute a ready-made pool of job candidates if a core paid developer leaves a project.

This developer community is a project’s best chance at sustainability, and collaboration at the developer level, rather than collaboration at the institution or administrator level, is usually key to a scholarly open source project’s lasting success. Getting provosts, deans, and directors from partner institutions to commit FTE’s and other resources to a project is very welcome—we’d love some commitments of this sort for the tool we built last week. But it’s not where the strength of a collaboration will be located. Individual developers, who commit their time, effort, ideas, code, heart and soul to a project, are the ones who will keep something going when money and institutional interest runs out.

A developer community does not develop on its own, of course. It requires support. First and foremost, a developer community needs open communication channels—an active IRC channel and listserv, for example—something which, in the case of a university of library-based project, means a group of responsive staff developers on other end. Community developers need profitable access to the project’s development roadmap so they know where best to contribute their efforts. They need well-documented and thoughtfully-designed APIs. They need technical entry points, things such as a plugin architecture where they can hone their chops on small bits of functionality before digging into the core code base. Most importantly, community developers need a sense of community, a sense of shared purpose, and a sense that their volunteer contributions are valued. All of this has to be planned, managed, and built into the software architecture.

This philosophy of use is core to CHNM’s vision of open source software for scholarship and cultural heritage. The tool the crew of One Week | One Tool developed—like Omeka and Zotero before it—should be case in point. It was chosen with clear audiences in mind. It was built on approachable technologies and engineered to be extensible. It’s outreach plan and feedback channels are designed to encourage broad participation. When it’s released tomorrow, I think you’ll see it is a tool to be used.

#oneweek #buildsomething