28:00 Anne Meeker
partly have a sense of care for the folks who are participating. We wanted to make sure that we were only talking to folks who had already formally separated from the federal government. We didn't want folks to get in trouble with their HR. This was explicitly not about whistleblowing. This was not about a space for confidential information. Like this was very clear to participants the whole way through. Like, this is public information. We will be sharing this publicly. We are happy to offer anonymity, and we are going to be absolutely uptight about making sure that anonymity is honored and preserved in our public facing formats here. But, you know, choose wisely for what you are comfortable sharing. So that was the intake form. That was the most important piece of the project. And then we wanted to give people some options for how to participate. Um, video participation was always part of the vision from the beginning. You know, we also recognize that folks were going through a lot and we wanted this to be as heavy a lift for folks to really get stuff off their chest or as light a lift for something you could do in two minutes on your phone while you were taking your kid to school. Um, we wanted to make sure that it worked for everybody in all those buckets. So we had the video option through TheirStory for the self-service video. Uh, it was kind of the self, the self-directed interview. And then we also had the written form. Um, and on later phases for the project, we ended up introducing two other options as well. We, um, did a small number of small group interviews where we'd bring a group of federal employees who had served together or served in the same agency together for a joint interview. And that was especially for folks who were particularly cautious about being very anonymous, who wanted kind of that buffer of being in a group of other respondees as well. Uh, and then we also ended up doing a small, very small number of one on one interviews for folks who had challenges with the tech or just really, really deliberately wanted like a conversation, uh, instead so folks could contribute. And then we did, uh, for folks who were in our pipeline who'd done the intake form, but hadn't like finished their submission yet. We did a lot of, uh, continual touches just to say we're still here. We understand there's a lot going on. Here are your multiple options. Uh, we screened, we independently screened on our site, all of our submissions for stuff that didn't, that we were not this project was not designed to take on. So we were screening for PII. We were screening for personal ad hominem attacks. We were screening for, again, that confidential information stuff that felt like it may want to be more of a whistleblower mode. And then we were screening for, content that got too political, which we, I think hardly had to do at all, which was very, very nice. Um, and then we ended up publishing our findings in a few different phases. So after the first big wave of the project, we did put up our first interactive archive so that, um, additional participants could see what it looked like. Um, and then we did our full final report with our interactive archives, our key findings report in our project methods report so that people could try to replicate it as well.
30:52 Zack Ellis
Wonderful. And I, I'm curious, um, here to dig a little more into that phased approach that, that you took. Because I think that was something that was interesting to me was how you didn't just launch big right away, but kind of learned incrementally to shape what the project would ultimately look like.
31:16 Anne Meeker
Yeah, definitely. And this was a really key piece of working with our, with our coalition and with that kind of formal coalition. And then the next level out of the kind of friends and friends of friends of the project that we wanted to make sure we were getting this right. Um, like I said, like we talked about a second ago, we weren't sure if folks were going to be comfortable participating the way that we hoped that they would. So we wanted to do our kind of initial friends and family rollout. That was a very, very small group of folks that we, we knew and trusted through our networks to say, hey, are we on the right track with this? Like, do our questions make sense? Does the process feel like it offers the level of anonymity and protection and trust that you would want? Are there better ways that we can think about our service delivery, our service design, for lack of a better word here. Um, and we got some fabulous feedback from those folks and we're able to kind of tweak a little bit and make sure that it worked. And then yes, we did our big, big launch and outreach, uh, through all of our coalition partners and their networks. And this was all kind of trying to dance around the, like the, the times and dates of the federal workforce stuff. So we knew that we a lot of the folks who'd taken the deferred resignation package, the fork, were going to process out of there. We're going to formally separate at the end of September. So that was kind of the date that we had in mind after phase one, that we knew the folks who had just immediately been been rift or fired or just left in other ways could participate through the fall, but or through through the summer and into the early fall. But we knew like the big bucket of folks was probably going to be in September. So that's when we we tried to get ready to kind of launch that second wave ended up being a little closer to November. Um, but that second wave of outreach trying to reach, again, the folks in that, that second wave of the deferred resignations.
33:05 Zack Ellis
Yeah. Interesting. Um, one other thing that came to my mind, just in terms of how you really paid attention to, uh, what I might call like the customer journey, um, it's just the people who are going to be sharing their testimony. You were really thinking a lot from their lens of what the experience would would be like. And even people who didn't finish the intake form, how to follow up with that, like all of that, um, what did this require from a capacity, from a staffing standpoint for you or for, for the project as a whole, as other organizations might think about if they wanted to do something similar, what, what would they need from a capacity standpoint?
33:49 Anne Meeker
Yeah. So on the, like just running the project for the most part, uh, we had, I'm going to ballpark it at about a quarter of my time. Um, and then once, once our, I think she's on here, once our fabulous program associate got started. Caitlin McNally. Caitlin. Thank you. Uh, once Caitlin came on board for our team, uh, in the fall, I think we probably had maybe a third. Caitlin can correct me if I'm wrong. Uh, third quarter, quarter to a half of Caitlin's time as well. So altogether, this did not add up to one complete staff member, and I think we probably could have dropped that even further. We we talked a lot at the beginning of the project. And, and as we kept going about, you know, how do we be ready if this takes off in a way that we're not expecting? Like if this was suddenly thousands of inquiries or thousands of submissions, we probably would have kind of reconfigured our process to do a lot more automation. But as it was, you know, this was a fairly manageable flow of folks who were coming in through the form, submitting those testimonies so that we could screen those testimonies and add them to our kind of clean, available for public public consumption spreadsheet. So it was possible to do it in a pretty manual way. We could have automated it, but we didn't end up needing to. So the kind of capacity that might take is probably a bit elastic.
35:10 Zack Ellis
Sure, sure. And, and for what you were trying to show around like this is possible. It was enough to go through the end to end, just process of how, how do we, uh, source, uh, uh, stakeholders to share their stories? How do we build the partnerships and alliances to reach those individuals? You know, what are the methodologies for capture? And then there are other tools as well. Maybe you could talk a little bit about some of the other, uh, technologies that you used as a part of, um, the whole project. Yeah,
35:46 Anne Meeker
Yeah, definitely. So, um, thinking about kind of the stack and again, the, the manual pieces of putting, putting all this together was, um, and coming back to the kind of participant experience here, uh, when you came to the site was hosted on our, on POPVOX.org website. So you went to the main project page and then we, uh, were a Google form team. So we ended up just using a Google form. Again, we were not looking for stuff that was going to be sensitive enough that we felt we had to build in a lot of specific cybersecurity around it. So that was kind of a self screening. We we told folks this was supposed to be public information so that we, we didn't have to get crazy in the weeds on building out something that was able to handle like confidential and sensitive information. So Google forms and sheets, everyone's used them. And it was also easy for our, our participants to use. Um, and just available for our full team. And then obviously we use TheirStory to capture the video recordings, which was phenomenal. Um, so that again, folks would, it was as close to the experience of being interviewed as possible without having a human interviewer need to take that time. Uh, which was amazing. So we are tracking stuff in our, TheirStory archive in our Google form. And then the other tool that we used, um, again, thinking, putting our congressional staffer hats back on, of trying to think about that question of, you know, if I get 100 responses and they're all long form written responses or they're videos that might be anywhere from 2 minutes to 2 hours, I can't just drop that on a congressional staffer and say like, please have fun and use this archive. Like, I need to be able to create something that makes this accessible. Um, so we, uh, I was familiar with talk to the city from, um, from some work that we'd done on just looking at the scope of AI tools available for public policy. So talk to the city is a very cool tool that kind of can bring together written and video responses and synthesize them into themes. And then from those themes, drill down into specific claims that are all backed up against those written, those written submissions or the video submissions. And so we worked with talk to the city to create kind of some interactive archives that gave people some, some hooks, some conceptual hooks on where to start in understanding what was in this archive. Um, although I think TheirStory has added quite a few features that would have let us do most of that native within TheirStory. Since we started the project.
38:08 Zack Ellis
We have, we have, we have expanded. And it may be if, uh, if we have time, we can, uh, show some of, uh, Of what we built together just over the last few weeks.
38:20 Anne Meeker
It's amazing. I think we should definitely do that.
38:22 Zack Ellis
Sounds good. Well, we'll we'll get to that for sure. Um, so as you were collecting, I'm curious, um, any surprises or lessons learned from, from that just collecting process. Um, main points of friction, any challenges that you had to overcome, just kind of in the midst of doing the, the collection and synthesis that, that type of thing.
38:51 Anne Meeker
Yeah, definitely. Um, the biggest one is just the most critical piece of all of this was the, was the trust and was the networks with our coalition partners to get this in front of the right people. So I think anyone thinking about replicating this, it is not as simple as if you build a really cool collection setup, like people will come like it is. The biggest piece of that work is getting in, getting onto the screen of the the right person at the right time with the right vector that they trust. So that was truly where we could not have done this without our coalition. And where if I could do this all over again with more staff and more time, we probably would have done quite a bit more outreach and a lot more tailored outreach, targeted outreach, maybe even like some paid digital stuff to try to get it in front of more people. Um, the next thing too, that I think I wish we'd been able to do differently, but it was a tension between trying to do that phased, iterative approach to make sure we were doing it right and being able to move quickly enough that we could capture the moment. I think if we'd been able to launch the first phase, like the first public phase earlier in the summer, we probably would have gotten more submissions. Just so that was the the big moment. It was Doge was happening. It was the riffs were happening like it was just a lot more salient in people's minds. And also, you know, our our participants here, they're moving. A lot of them were moving on. Um, they got fired from their federal job. You can't stay unemployed for very long in this economy. So I think we had a window when we could have caught like when we were catching folks, but we probably had a tighter window to catch folks before they really moved on, you know, started forgetting some of the things they'd wanted to share. Um, so if we'd been able to move faster, I think we probably would have, would have been able to up our participant numbers a little bit.
40:32 Zack Ellis
Sorry to interject. What do you think would have let you move faster?
40:36 Anne Meeker
More time, more staff time than trying to do this as, as kind of a, like I said, a one of the 4 to 5 major projects on my plate. Um, and just not having dedicated staff, uh, you know, and again, this is all this is all on us. Um, our coalition partners were amazing. And the delay, any delays were 100% on our end. And they, you know, we built in those constraints for a reason, but having more people to move faster, I think would have been, would have been interesting. Yeah. Um, the last thing too, I wish I'd reconfigured that anonymity, uh, sheet because it was, as I said, the most important thing, the thing that we spent most time on making absolutely positive that we honored folks requests for anonymity and anonymity. We only we only identified folks by the specific pieces of their information that they gave us permission to do, which was ended up being really complicated to just track through and even kind of screening within some participants responses. So if they said, you may only refer to me by first name, and then they said their last name in a video, or they gave me a piece of identifying information in their written response that made it really easy to figure out where they worked when they didn't give me that permission. That was a lot of kind of hand combing to make sure we were all good there. So there was probably a better mechanism to do that that we could have found, but we made it work.
41:54 Zack Ellis
Yeah, absolutely. It sounds like it was at the very least, that attention to detail when it came to anonymity, while maybe painstaking, also helped get more respondents.
42:07 Anne Meeker
Yeah, I think so. We did answer a lot of one off questions from our potential respondents about like how all this information be used, how are you storing it? And we did keep coming back to, you know, we are doing everything that we can. We our first value here is making sure that we honor your anonymity request. But this is public information. Expect that it will be read widely. Expect it will be read widely on both sides of the aisle if we do our job right. So be cautious.
42:30 Zack Ellis
Yeah. Talk a little bit about, um the why video was important at all. It sounds like written was probably the easiest way to ensure anonymity. Why do video at all and what was the role of those responses?
42:45 Anne Meeker
Yeah, definitely. So we really wanted video for so many reasons. Um, a big one first was just ease of and ease of submission that again, this was our boss at the very beginning. I remember we were in like a project planning meeting and she said, like, if I'm a departing employee, like I don't want to give any more time here. What I want to do is just hold up my phone and record myself again in the two minutes while I'm doing school drop off and then bounce. Um, so we definitely wanted video for just that ease of use. But then also, I mean, if you've looked through our archive, the written responses are incredible. Like there's so much richness and value in them, but also, but particularly in the video ones, you just have this sense that you know someone in a really different way from seeing the video than you do in a written response. And again, thinking about our, um, our audience here again, I, I'm, we talked about my constituent service work. Like part of why I was so lucky with that work was that they were individual stories. And I still remember faces and I remember names and I remember anecdotes. I remember the constituent who raised German shepherds, like that's in my head forever. And we wanted to make sure that and we wanted our, our archive to be as, as accessible and usable to, to staff, legislative staff as we could do it. But we also wanted to make sure that it was memorable. And so I think giving legislative staff
