January 2010 Site of the Month
January 2010 Site of the Month
APA Exposed: Everything You Always Wanted to Know about APA but Were Afraid to Ask
Authors: Deborah Garson, Wendy K. Mages
Interviewee: Deborah Garson
Institution: Harvard University
Interviewer: Genevieve Williams
Q: I’m speaking with Deborah Garson about the APA Exposed tutorial. Deborah, I’d like to start by asking what prompted you to create the tutorial, and what purpose does it serve?
A: Overall we’re proactive here on information literacy, and involved in promoting academic integrity and preventing inadvertent plagiarism. So, to that end, we’ve focused on having information literacy as a key in all the workshops that Gutman Library has done for a very long time. But probably ten years ago, we became very aware that our students were struggling with the APA format, and the rules and citation information. And APA is required, pretty much, here at the Ed school. Not across every single class, but I would say 90-95 percent of the instructors use APA, so it’s an important component of our students’ work here. We found a graduate student here, her name is Wendy Mages and she is the co-author of the APA tutorial. She had become really quite expert at interpreting the APA citation rules, and knew the manual inside and out. I talked to her, and she agreed to do some on-campus, on-site workshops. So we started those, they were over-subscribed, they were always packed, and we realized that we weren’t reaching everybody who needed APA assistance, even with the on-site workshops, obviously, at two a.m., if a student had a question the APA workshop wasn’t happening. We had some students at a distance, and we decided that technology was the perfect thing to consider for this. So we sat down and we talked about it quite a bit in the reference department staff, we talked with Wendy about it, and then we involved our learning technology staff upstairs, with thinking about how do we transform a very successful on-site classroom workshop into something that’s just as successful as an online tutorial? So that was the impetus behind it: we weren’t reaching as many students as we wanted to. And this way it was going to be available 24/7, so if it was two am and the students had a question, they would be able to refer to the tutorial. And even if students took the workshop, and they still had a question, simply because they couldn’t remember something, or a new format appeared and they weren’t sure, well, do I interpret it this way or do I interpret it that way in order to cite it.
Q: So you’ve got some indication that even after students have done the in-person workshop, they come back to it for review?
A: Well, it is for review, but I think it’s also point of use. I mean, in the workshop, Wendy’s emphasis was understanding why one would want to cite to begin with, and the importance of it, and then the how to. But it doesn’t always mean that the student walked away saying, “Oh, when I write this paper for Professor X, I can remember now how to cite the PowerPoint that he referred to,” or something like that. Initially we thought of it as supplemental, but it became very evident as we went to develop it that it truly could be a standalone, and that’s the second version that we’re finishing up now, because APA came out with the new edition in July. We just reworked the whole tutorial, and we’ve been continually working on it because we have feedback built into the tutorial when anyone does it. And it’s open, it’s not restricted at all to Harvard.
Q: It’s been really great to be able to show it to our students, and say here, check this out.
A: Well, it’s amazing—I think that’s one thing we were absolutely not expecting at all: the number of hits the tutorial has received from non-Harvard users. That was an unexpected and wonderful kind of surprise. And we have gotten e-mails. In fact, when we were beta-testing the online tutorial, in the last seven months of the beta-testing period it got over 7000 visitors. It’s been a significant resource for people in the education community, well beyond just Harvard using it. That was pretty amazing and really nice. And the feedback we’ve received: student e-mails and instructors’ e-mails, even some high school students who were required to use it said this is great, this is wonderful. It definitely tied in with all the homework we did about how to create a really effective online tutorial, best practices around it, and things like that. We’re pretty pleased with it. But it is a lot of work.
Q: I can imagine. Can you talk a little bit about some of the background research you did in preparing for this, some of the materials that you used? Not necessarily specific citations, but how you went about doing that kind of background research?
A: We looked a lot at the literature around online tutorials, and it was a mixed bag. A lot of the articles talked about the assumption that you can just take paper and paste it up on the Web, that a successful workshop that is taught in a classroom is going to translate just as successfully to an online environment, and that just isn’t true. Obviously. Because the big missing piece with any of that is the facilitation that goes on with an instructor being there, so you’re missing body language and you’re missing other things that really help students stay engaged. And I know, from watching Wendy in the workshop that she did for us, she was very—people just walked out of those workshops saying she’s so great, she’s really a tremendous instructor and I was thinking, oh my gosh, we’re going to lose all that when we go online. So we spent a lot of time working very closely with the learning technologists upstairs, that we luckily had a great working relationship with and they were very interested in doing this, and extremely helpful when it came to developing best practices around it. And then we looked at how to make it as interactive as possible, because again we’re missing that in-person instructor or facilitator giving you some clues or feedback. We wanted to make sure that there were multiple means of engagement, so we used a very flexible user interface. And that means that users can elect to skip, or to replay certain topics. They can choose to use the learning checks or not, you know, obviously maybe once they’ve been through it they look to where they need to go just to refresh their memory or something. We built in the search box, and we wanted to make sure that it would work for different learners. So there’s voice in there, people can read the information, they can listen to it, so there’s a lot of different ways for people who have different learning styles to access this information and walk away with it. That was really important for us, making sure that was built in. We looked at and integrated the Principles around Universal Design for Learning, which is about offering multiple means of representation and engagement when you’re building something.
Q: Initially, who was your target audience for this tutorial?
A: It was definitely our students—graduate students at the school of education. We were continuing to offer the on-site, in-class workshops, but we thought, this is for the students who can’t come because they have schedules that conflict with the workshops, so they can’t attend them. Or again, that it’s two o’clock in the morning, or it’s a Saturday, they have a specific question, and the paper’s due on Monday. So initially, that was it. But as we got further into developing it, we made an administrative decision to open it up to the public to beta-test it, and that’s when we realized that this went way beyond us. Even though there are components that are specifically focused for our students, at the same time we realized that the basic principles easily translate into many other disciplines, not just education. It definitely translates very well into many of the other social sciences.
Q: You mentioned that students have used this tutorial after a library session. Have you had students come to a library session who have done the tutorial already?
A: I think we probably have, though I don’t have the statistics in front of me for that. We wanted to see how it was being used. And it was out there both on the external web and on our internal intranet for our community. So I’m sure that students were actually looking at it and then coming to the workshop. And one of the questions that we handed out on the feedback for the in-class workshops was, have you used the tutorial? So I think that some students had actually taken it. But Wendy has such a phenomenal reputation as an instructor that I’m sure that even people who took the tutorial said, I still want to go see her in person. And it would all depend on the timing; they might have assumed that they couldn’t make it to a workshop, and so they did the online tutorial and then discovered that their schedule opened up or they were able to attend one of the workshops.
Q: What are some of the ways that you’ve incorporated core concepts or best practices pertaining to information literacy into the tutorial?
A: We really wanted to build in a flexible user interface. Also building in the learning checks, so that students themselves could check to see, did I really understand what I just went through on the tutorial? And originally the tutorial was at least 40-45 minutes long, the first one that we beta-tested. So then we knew we needed to break it down into modules. Which then means that students could go through it once, but then maybe they’re going to realize that they need to just go to the second module because the information they need is there. Wendy was very good about—more than good, she really was very knowledgeable about how to present the information. She built in a lot of examples, and she’s very good at explaining without making it seem too overwhelming—why you would cite the author this way, and how to interpret the rule that way, so a lot of the tutorial is filled up with making an instructional point, but then backing it up with specific examples that reinforce it. So somebody may not remember the exact rule, but chances are they will remember the example that was used: “Oh yes, I remember that three authors have to be done this way,” or “I remember now that I do have to cite a PowerPoint presentation this way.” I think that trying to make it manageable in terms of breaking it down into modules was also very important, because otherwise it can be overwhelming: somebody’s going to say, “I can’t sit through sixty minutes,” but they could say, “Well, I’ll do the first two, and then I’ll maybe do the next two tomorrow,” or something like that. Or, “I’ll do the sixty minutes the first time because then I’ll know where to find what I need when I’m looking for something specific.” We did do a lot of beta testing with it, and got a lot of feedback about people making substantial improvement or some improvement after looking at the tutorial. So it seems to address the needs of both a novice and a more proficient APA user.
Q: Can you talk a little bit more about who was part of the design process and who you collaborated with, aside from Wendy? And how did you all work together?
A: It was myself, administratively and also because I had put the workshop together with Wendy. She was responsible for the content and I got the approval to have her be the instructor for these workshops. Together we built the feedback survey for the workshops that were held on site, because Wendy wanted that feedback. I wanted it, too, to justify continuing to have her as an instructor. When we reached the point of realizing how wonderful the workshops were, but were still having students coming back saying I missed it, or I forgot, or something like that… I had been looking online for awhile because I was very interested in online learning. I had taken a class here on online learning, a graduate school course, in the hopes of thinking about what we do in Reference Services that could be translated into an online tutorial. So these things just kind of meshed, and I thought, well, this is something that I think is doable. I think I didn’t realize initially how much work it would be. * laughs * But, you know, I don’t know if anybody ever does. So Wendy and I sat down and I talked to her about it, because I wanted to see how she felt about doing it. Because she was going to be the voice, and her content was going to moved into an online tutorial. She was extraordinarily enthusiastic about it, and has worked so hard to make this tutorial what it is, and continues to be heavily invested in it. Then we had a meeting with the learning technologies staff upstairs, who were very enthusiastic. One of them was a graduate of the Ed School in the Technology and Education program. So she really helped endorse what I had learned in my online learning course, about building best practices into any kind of online—whether it’s a tutorial or whether it’s something more extensive. So it was really a series of meetings, and figuring out what was going to make the most sense in terms of software and hardware, and could we afford to do this or was it going to be free, and so on. So it took awhile, the meetings, but everyone was 100% behind this project and everybody was very enthusiastic about doing it.
Q: When it came to choosing not just to do this in an online format, but the specific presentation technology and media that you used, what were some of the key drivers of what you chose to use?
A: First and foremost was the LTC’s recommendations, because they were familiar with certain types of software versus others. Secondly was cost, because we certainly couldn’t afford—nor did we especially want to—hire an outside software person or consultant to help us do this. And so, to some extent, our choices were limited by cost and by staffing. But I really trusted that their experience upstairs was going to work for doing this, because they’re very knowledgeable and they’ve done a lot of work around building online assets or products. One thing that we knew was going to be critical was being able to very quickly update this. And add to it, if a new format came along or a student said “How do I cite from Twitter?” or something like that. We wanted to make sure that that was built in. The LTC investigated software and came back with very strong recommendations. We trusted their judgment. None of us have been disappointed at all, it’s worked fine. It also has to do with the equipment that’s available to do the recording, and so forth and so on.
Q: What were some of the technologies you used to create this tutorial?
A: We used Articulate, and then for the audio we used Audacity, and then of course there’s PowerPoint in there, too. And then we have a recording booth upstairs in our building, that’s part of the Learning Technology Center. That’s where Wendy recorded the content for the tutorial.
Q: How easy was this software to work with? I imagine that some of the people who are going to be reading this are thinking about creating something similar. So was it fairly straightforward, or…? How did you find that experience?
A: The advantage we had, and some people will not have this advantage, was having the Learning Technology staff available to us. They really did set this up and guide us through it. At this point with the APA online tutorial, we can do a lot of it ourselves, but thankfully they’re still involved. So they’re there to guide us if we have a question about it. There would be a learning curve. I think that Articulate and so forth is pretty straightforward. But again, we did have the particular advantage of having staff who already knew the software, and were willing to work absolutely side by side with us in the first building of the tutorial.
Q: About how long did it take you to put the tutorial together?
A: I think it probably took us six months, from the very beginning of sitting down with Wendy and talking to her about doing it. Now some of that is based on her availability, because she was a student at the time—she is graduated with her degree from here. And some of it also had to do with the Learning Technology Center, they had a lot on their plate. So I would say that it was probably six months, but again, you have to factor in that some of that was when we could squeeze it in. And the initial learning curve. Then we beta tested it for almost seven months. So it took awhile, it really did. Having said that, I think we’re all just delighted to have such a strong product out there. But even the revisions have taken us—let’s see, the new APA came out July 15th, and we’re just starting to finish up some of the revisions in the tutorial for the new release from APA. But nobody here is dedicated to it; I don’t have a full-time person here who’s doing this.
Q: So it’s something you have to do as you can make the time for it.
A: Yeah. It is a priority, and as much as possible we have put other projects on the back burner for the time being, because we know how important it is to the community here. But again, it is taking away from other time and other responsibilities for it. And since Wendy is no longer here at the Ed School, she’s at Boston University now…you know, she’s just been a tremendous trooper about coming over from Boston University to re-record, and to help us interpret some of the new APA stuff.
Q: You mentioned that you did about seven months of beta testing. Did any of the feedback or results you got from that lead to changes in the tutorial’s presentation?
A: To tell you the truth, no. I mean, there were some small things, like somebody said something about the typeface, and maybe there were one or two typos that got by us. But overall, no. The initial one we beta-tested did not have modules, so there were one or two comments about it being too long. But at the same time one of the comments said, “It’s so long…but thank you so much, because I learned so much from doing it.” So, you know, there were some individual comments that we didn’t feel necessitated going back and making any changes. Other than we did know long-term that we wanted to break it down into modules. That had been a recommendation from LTC to begin with, but at the time we all felt that we had a schedule to get this out, and we wanted to stick to that for the first round. And then we could go back and break it down.
Q: Have you done any kind of formal assessment of the tutorial, or do you have plans to do that?
A: We have done some, and actually Wendy and I presented at AERA, the American Educational Research Association. We also have an article coming out that does talk about the kind of feedback that we did. [That article is now available. – GW] We looked at the effectiveness of the tutorial. Overall, the majority of the participants, like 79% I think, reported that they found it very useful, and another 19% reported that they found the program useful. We had a survey at the end of the tutorial, an evaluation survey, and we collected sample demographics about people who were using it, and who was using it, and more importantly, how useful they find it. And that’s where we began to see that even people who thought they were familiar with APA were saying that the tutorial was helpful for reminding them about certain things, to be able to quickly refer to something that maybe they thought that they knew but they weren’t quite sure about. So it was basically just an online survey that we created to evaluate the tutorial’s quality and utility.
G: What means of discovery do you have for users to find it? Is it linked from the library website? What are some of the ways that people can get to this information?
D: When we first released it, we did a huge big PR push. I’m a member of SLA, and I let the Education Division know. Also we contacted other education and library associations such as ACRL, EBSS, and so forth. That’s where we wanted to target it first, mainly because it was coming out of the library. I wanted feedback on it from the library community, as to what they thought about it. Since then—Wendy and I did a presentation at the last AERA conference in San Diego, about the tutorial. And if you Google it, it definitely shows up. So I think we’ve done, or tried to do fairly extensive PR for people. Especially since it is open and there aren’t restrictions to people using it. And I would say, given the number of people that have used the tutorial, or at least the number of hits on it; we’ve even had people internationally using it. Canada, Australia…so people are finding it.
Q: Are there ways that you were publicizing it to your student community?
A: For the student community, we put it up on the library’s intranet, which our students all go to and use. We have monitors in the buildings on campus, and it was put up on the monitors—and still is, because we remind students about it, and obviously we have new students come in every year. We sent out e-mails to specific segments of the community here, such as doctoral students, master’s students, the faculty. I made a presentation to the faculty at a meeting. I’m also in charge of writing services here. I’m always somewhat surprised when somebody doesn’t know about it. I do a pretty extensive repeat every year of the fact that the APA tutorial is there, and obviously reference staff are doing presentations, whether one on one or to a class as a whole.
Q: You mentioned earlier that you don’t have the numbers in front of you, but are there general trends that you’ve noticed in the usage statistics, or anything that jumped out at you or was surprising?
A: The outside use just absolutely took us by surprise. And there were some months where maybe the statistics went down a little bit, but then popped way back up, you know, in September or October or November. That was typical, that was kind of what we were expecting anyway. But for sure the number of hits the APA took the first year or two was extraordinary and not expected. And the other thing I think we were hoping for, but maybe not anticipating as strong a response, was even users who felt they were familiar with APA said they found it very useful or useful. So that was really helpful as well.
Q: What was the biggest challenge you encountered in creating the tutorial?
A: There was a challenge in wanting to absolutely ensure that this really successful in-class thing translated just as well to an online environment. Having said that, I think that we absolutely did our homework, in working with the Learning Technology staff here, I think we were successful. I think what none of us was anticipating—we were anticipating it but not to the extent that it required—was the amount of work that it does take. Sometimes you have the impression that people can put these little online guides or online tutorials together very quickly. This wasn’t the case. And it might have had to do with the fact that this was a fairly sophisticated thing to move into an online environment. I think we really had very high standards for it, and we did not want to discover that we were just sticking something up on the Web that helped one or two students, but for students who didn’t learn as well just looking at text, or students that needed Wendy’s voice there, or something—that we hadn’t addressed those issues of different learning styles. So would I do it all over again? Yes, I probably would. But it’s labor-intensive. I think anything worth doing is labor-intensive.
Q: What advice would you give to someone who was considering creating a tutorial like this one?
A: Collaboration, first and foremost. I think sometimes we don’t realize the expertise and the knowledge that a colleague can bring to it. Or even a different perspective, you know. Once we developed it and had the reference librarians look at it, we had some undergraduate students look at it, and that’s all valuable feedback. Not that you want to spend six months having people look at it, and you don’t necessarily want to go back and totally revise it because somebody said the color of the background wasn’t quite right. But collaboration is key, and doing your homework is essential. The more you read and look at other formats and things…and I think that’s an important point, too, that the format that works for me might not work for someone else, both in terms of staffing and what my community expects, or what I can deliver in a successful way to my community. And I don’t necessarily think that online is always the answer. If there was a way to have Wendy available 24/7, I probably would’ve done that. But again, it’s harnessing technology to our advantage, as long as you’re doing it with best practices behind you. And I think a lot of people are willing to help out, depending on how you distill it. So part of it is being a very good salesperson for your own ideas, and saying, this is why I think this is important, and this is what I think it’s going to do, and it’s important because it’s going to revolutionize the world, or something. So even though somebody may at first be reluctant, they may say, you know what, I do have an hour or two that I can contribute, or something. And I wouldn’t reinvent the wheel. If somebody else has done something and you can adapt it, I say that you take it—with their permission—and work with it.
Q: Is there anything else you’d like to add that our readers might like to know?
A: No, but they should feel free to email me if they have questions. And the only reason I can’t give out some of the results from our feedback and evaluations is simply because it’s coming out in an article. But I certainly would share it with somebody over the phone. And I really do want to give huge kudos to Wendy in this, because she is absolutely the voice of the APA online tutorial, as well as its heart and soul.
Q: She did a ton of work and really made this happen.
A: She did, and she made it happen even before we put it online. And that’s another place that I think I lucked out. But again, you know, you go out in your community and find those people, because they’re there. And that can help all of us in terms of information literacy in our communities.
Q: Thank you so much for your time today.
January 2010 PRIMO Site of the Month