AMIA AI Webinar

0:00 Jenni Matz

Um. Hi. Oh. Thank you. Zack. Um, welcome. Uh, my name is Jenni Matz, and I am the co-chair of the AMIA Oral History Committee and the AMIA Copyright Committee. And this was a topic that seemed to cross both interests. So, um, we extend a hello to whichever committee led you here. Um, and we're excited to talk today about AI, the perils and the power of AI with Zack from TheirStory and Caitlin from the 92nd Street Y, uh, who are going to introduce themselves in just a few and talk about the project that they worked on together. So just some housekeeping. Um, I'll ask you to just add your to mute yourselves and add your questions in the chat, which I will do my best to monitor while Caitlin and Zack talk. And then we will leave time for Q&A. Plenty of Q&A at the end. I'm sure everyone will have some questions. Um, and so, uh, I think everyone in the greater AMIA community, you know, we love history, we love accuracy, uh, and we love to find new ways to discover and create access. Um, and, uh, other access points to our collections and make them as accessible as possible. AI, of course, is transforming all of that. Um, and the way that we collect and process our collections. So there's some amazing tools, which I know many of you have been experimenting with that we can now use to enhance discoverability. Uh, but there may be pros and cons, um, with this technology. So today is just an open discussion about some of the issues that impact the audio visual world and, um, our day to day workflow. So both of our speakers today have spoken on previous panels for the Oral History Association. Um, if you are part of that organization or a member, I encourage you to definitely check out some of those past webinars. Um, and OHA has done a lot of work in this area as well. Uh, the goal here today is to share some of the lessons learned from a project that use AI for transcription and indexing a collection, and hopefully spark a larger conversation on the issues related to AI and IP, AI and privacy, informed consent, etc., that touch on some of the legal and ethical issues with collections. So I will now throw it over to Caitlin Biggers from 92nd Street Y going to introduce her project.

2:49 Caitlin Biggers

Hi everyone. Great to be here with you. Um, I just want to apologize in advance. Our building is under construction, which is great for us, but you might hear some drilling sounds, so my apologies if that happens. Um, I have been the archivist here at the 92nd Street Y for the past three years or so. Basically, I oversee the organization's vast archive, and my primary goal is to try to get as much of our collections into the hands of the public as is possible. So when I joined, I was pretty fortunate to jump right into an AI transcription and named entity recognition project that was really designed to make some of our most important recordings more easily accessible. Um, just to give you a bit of background on kind of the material that we're working with, uh, the 92nd Street Y is a cultural and community center in New York City. We have a more than 150 year long history of supporting arts and culture programming, amongst lots of other functions, like we have a nursery school, people live here, there's a gym. Um, it's it's a really wild place to work. Um, but the Unterberg Poetry Center is, uh, one of our most prominent programming areas, and it holds an incredible collection of event recordings dating back to 1949. Um, these recordings include readings by huge names, people past and present like Truman Capote, Kurt Vonnegut, Joan Didion, Mary Oliver, Susan Sontag. It's really like a who's who of of literature in the 20th and 21st centuries. So it's, it's a really exciting collection. Um, all of the recordings were made at nine, two and Y and typically feature authors reading selections from their work, sometimes solo, sometimes in conversation with another performer. Sometimes they're debuting something, sometimes it's a passage from one of their most famous books. So it's a good variety. Um, if you could hit the slide. Zack.

4:48 Caitlin Biggers

Um, we, we knew that we wanted to transcribe the recordings because transcription instantly makes everything easier to access when you're talking about audio. And this was especially important for us because most of the recordings are an hour and a half long. So being able to trim that down and find the piece that you're looking for is hugely helpful. Quickly, it became pretty clear that time and consequently, budget would be the biggest factors in shaping our project, because we were looking at almost 3000 hours if we were going to do the entire analog collection. So from a grant term and budget standpoint, we it was clear basically from the beginning that we needed to use AI. It was just too much material to do manually. But still accuracy was very important. I mean, these are these are poets and authors and writers talking, you know, reading their work. We can't have AI like putting words in their mouth or kind of like trying to say what they kind of meant. Um, so transcript. Authentication was a part of the conversation at nine two and why right away? Ultimately, we decided to transcribe and authenticate a thousand recordings, which is about half of the analog collection. Um, and that's about 1500 hours. And we also chose to apply to LM based tools, specifically AI indexing and named entity recognition to increase subject searchability and just generally make the collection more usable. You could do the slide deck. Um, basically the final product, each recording is accompanied by an index that breaks down sections of the transcript by topic and allows users to jump to the relevant section of the recording or the transcript. So rather than finding things like, you know, introduction, author remarks, um, it'll actually give you a little bit of information about what the speaker is talking at a certain point in time, talking about at a certain point in time. Um, and you can see an example of that. Just a screenshot there on the right. And the next slide please. Um, we also built a named entity recognition or NER pipeline, which I'll let Zack talk about in greater detail. But generally for those who aren't familiar, NER is a large language model based technology that enables text to facilitate data analysis. And I've got a view of that here where name terms are highlighted in different colors. Um, to me, this, I mean, any, anything that gets trained material transcribes and makes it easier to use is huge. But to me, the indexing and any are pieces of this were some of the more exciting parts of the project. Um, but they were also more challenging because they brought up the most things to think about when it came to the application of AI and the potential ethical considerations. Um, and ultimately those elements are what we're here to talk about. But first, I want to turn it over to Zack, who can speak more to their stories perspective on these tools and challenges. And we worked closely during this whole process. So take it away, Zack.

7:53 Zack Ellis

Absolutely. Thank you. Caitlin. Um, so I'll dig more a little bit into the specific tools and uses of AI in the project and show, um, the end products that we've had some screenshots, uh, here on in action. So you can kind of see how it actually works together, uh, in practice and we can share the links so you can explore them as well. Um, Jenni did ask me to give a brief overview of TheirStory just so you have some of that context, and then we'll get specific to, to this project. Um, so I think, uh, it is worth sharing just a little bit of the history of TheirStory. For those of you who have heard me present before or talked to me one on one, you've probably heard this story, uh, about the founding of TheirStory. But for those who, um, haven't heard about TheirStory or, or, uh, or we haven't met a little bit. Um, by way of background is that I started TheirStory, um, for very personal reasons. Uh, and I used to work in Silicon Valley in deep cloud infrastructure as well as home health care and math education software. But, uh, it was really the personal side of things that got me into this world of oral history and archives and metadata and all of this. And it was a catalyst moment, uh, in 2014 where I, while working in Silicon Valley, visited, um, Amsterdam and went to the Anne Frank house and there was a clip of an oral history, uh, that was playing, um, on loop and why it meant so much to me was that my grand grandfather was a Holocaust survivor. And my dad's biggest regret was never recording his father's stories before he died. And so it was with that context where I heard, um, a clip of Anne's father, Otto, uh, playing on loop in the attic, where he concluded that because he had such a close relationship with Anne, but never really knew who she was until after she had died. And he read her diary that he believed most parents don't know, really, their children. And so it hit me, well, if most parents don't know their kids, how could I know who my parents are? Uh. How how well do I even know myself, for that matter? And so I started a project, uh, recording my parents telling their life stories from across the country. They were in Rochester, New York, where I grew up. I was in San Francisco. Um, and it was really just a matter of my own personal interest to build a platform at the time. Back in 2014, before there was Zoom. You know, maybe there was go to meeting, but nothing really purpose built for this, um, application. Um, but built the early stages of a platform to record, preserve and share our own family stories. Now from that time, and really since April of 2020, when I started working full time on TheirStory, um, we now provide a software application that is used by over 120 organizations around the country, some outside of the US as well. Um, and everything from large research universities, uh, who use it for their own, uh, making available their own, uh, archives or for project based learning where students are conducting oral histories or even things like qualitative research, uh, to folks like the 92nd Street Y, the nine over 11 museum and small, under-resourced communities as well, like the Chickaloon native village of the tribe in south central Alaska, who, um, preserve their native language and gather elder stories at tribal convenience. Um, just a little bit in terms of context of the technology. Uh, and then we'll go into the specifics of the project is that for any organization that is looking to embrace the power of audio visual based storytelling, as you all may very well know, being members of EMEA is that you have to stitch together a lot of these different applications to go from making something that is, uh, whether it's an existing piece of content that you have, or maybe it's new recordings that you're making, but how do you get that into a system that you can then transcribe index? How do you manage informed consent? Um, how are you generating metadata? How are you making connections across that? How is it searchable? Uh, and then how are you making that available? And that becomes very logistically burdensome. And so, um, that's what we've built and that's was a big part of the project. Um, uh, with the 92nd street y is having a solution that that could make that all possible. Um, so everything from remote interviewing to taking existing, uh, archives and transcribing, indexing those and making them available. Um, so no matter the source, one place to manage that, uh, whether that is with, um, individuals who are then authenticating that work, which was prominent in our project with the 92nd street Y. Um, and then making that available across different platforms. So that was, um, a little bit, just to give you a sense of what is TheirStory, but let's take a closer look at, uh, how this worked in practice with the 92nd Street Y. So I'm going to share my screen here.

13:26 Zack Ellis

So as Caitlin had shown, come on screen share. My Zoom screen share has frozen. Okay, there we go. Can you all see my screen now?

13:42 Jenni Matz

Yes. Great.

13:42 Zack Ellis

Great. Um, so as, uh, Caitlin had shown, uh, what they ultimately were able to produce was, uh, both a public site on their own. Um, uh, website here where you can browse the, the collection of recordings, about 850 that were ultimately made, made public. Um, and if I go into any of these recordings, we have the, the output of the actual recording. Uh, we have chapters, uh, that have been created and we have the interactive transcript, uh, all of which is, um, uh, Ada compliant. Um, and that just makes it much more accessible. But we were also able to develop what we called a research portal. So a way to actually have cross searchability. Um, ultimately now this was something that we developed together with a gentleman named William Mattingly. Uh, William was one of the lead, uh, data scientists at Yale. Um, and, uh, informally at the Smithsonian. Uh, also does a lot of work with the United States Holocaust Museum on using AI and machine learning for, um, sense making and generating metadata, uh, across their different, um, uh, sources from images to documents to audio and video. Um, and so what we had to do was to create first, uh, the methodology for transcribing and, um, and editing that material. So I'm not going to show the, Y's back end. But just as an example, um, we had to first build an integration with box.com, uh, because the Y had 1000 recordings in box.com and just the manual process of migrating files from one system to another would have been incredibly burdensome. So that was the first step. Um, as, as those materials were uploaded, they, uh, could all be automatically transcribed. This is actually the transcript of the Oral History Association presentation that we did together. And then the audio transcription center, now the Sicilian Precision Group had, uh, their team of Authenticators come in and actually manually edit the transcript, um, detect speakers, um, and also edit the index that was automatically generated. Now we used whisper, uh, as the speech to text, um, uh, algorithm in the background, uh, which we self-host on our own AWS infrastructure so that the data stays private. Um, and then the indexing was done in this case through the GPT API, uh, where they also specify that through the API, there's no training that is done on the data, but we can get more into talking about some, uh, new elements that we're implementing to even make that more, um, uh, locked down. Um, so this was a huge part of the manual process was the authentication of this work. Um, and then we also developed our own open source, um, named entity recognition, uh, model, uh, together with William Mattingly. And one of the things, again, that was important, um, was the, the human side to all of this. Uh, it was a process of going back and forth with Caitlin and one of the PhD researchers that were on the project with the Y in terms of determining what were the the entities that we wanted to extract. Um, we actually went through a process of the researcher would correct the entities that, that were extracted, and then we'd retrain the model to try and make it more and more accurate. Uh, and then all of that came into one place where I can say, you know, show me any place where a person is referenced and where I might ask something like, what was the role of women in literature? And just like that, across those 850 recordings, we can get back relevant excerpts. Uh, and so here, this top, uh, excerpt says one of the themes that emerges in a lot of the not criticisms of your writing, just observations of your work is the role of the male in a lot of it. And of course, you've argued in The Color Purple that it wasn't really about men. It was about women. And so I can click into this and it'll drill into that exact moment. Um, so that we can more readily use the audio and video as primary source. Um, we can also explore, uh, by those entities. So maybe I want to find every place where Toni Morrison was referenced, whether it's in this recording, uh, or maybe it's across the entire project. And so here we can see other places where Toni Morrison was referenced and that quickly be able to navigate the, the collection. But it was that combination of AI that helped to save time. Uh, this was a project that was done in a, in a year. Once we started with the actual transcription indexing named entity process. Um, but human authentication was paramount, uh, to make sure that the information was accurate. Uh, and then putting that all together into one place to be able to to explore. So I'll pause. Pause here. But this was a little bit about the technology behind making the archive more accessible.

19:33 Jenni Matz

Um, perfect place to pause, Zack. And perfect question in the chat related to what I'm going to ask you, which is accuracy. So can you talk about what the limits are right now of AI and Y for this project? You can either of you can speak to this. Human authentication was necessary.

19:53 Zack Ellis

Totally. Uh, so the there's a couple things to note in terms of accuracy. Um, the timing of this project was, I'd say fortuitous, um, because just a few years ago, um, to be able to transcribe this volume of content in a year and to have the human authentication on top of that would have been incredibly difficult, if not impossible. Um, even though there were models like Google had, you know, the, the Google speech to text model and so on, what people would often find was that the audio quality had to be incredibly strong and very clear speaking and likely English, um, to really have a speech to text output that would save time to then go in and correct. Um, now a few years ago, that really changed when OpenAI released their open source speech to text model called whisper. Um, and this was a model that was way more accurate across multiple languages. And we had done testing previously with the audio transcription center, who were the, um, the group that was doing the authentication in this project where up until the release of whisper, they would always come back to us and say, Zack, it sounds promising, but it cost us more to actually authenticate the the AI generated transcript than to do it from scratch. Once whisper came out, we did that test again. And for the first time they said, you know what? This saved us time. Not on every single recording, but overall across the corpus of the content. It did save us time. And so this was the first project that we actually did together at a large scale, where it was the combination of AI and, um, and human authentication. Uh, but it was not perfect. You know, we had examples of, uh, the audio transcription center. Now Cecily precision Group like to tell the story of like one of the first two recordings that they transcribed was in Yiddish. And the transcript was so bad that they thought to themselves, oh gosh, what did we just get ourselves into? We have like a thousand more recordings to go. Um, and so, so it was not perfect that like, for example, that one, uh, they had to do from, from scratch. And then we actually built a tool that could take, um, a transcript that was edited out of our platform and uploaded into our platform and then align the, uh, the transcript, uh, automatically to, uh, to the word. So we built some tools along the way to help us, but it wasn't, uh, was not perfect. The other benefit of an open source model is that, um, especially when well adopted, is that it continues to get better. So there are, for example, new versions of whisper that weren't available when we even did this project, for example, for Yiddish, um, that are way more accurate than it was then. So if we were to do it again today, then we may have had less, uh, recordings that, that had issues like that, but, um, but it's still not perfect. Um, that's, that's for sure.

23:06 Jenni Matz

Right. Which is another reason the human authentication is, I think still critical for these. Um, I'm going to get to the LMS and the training models and the, what you mean by the closed server in a second. Yeah. But, um, for, for just to, just to back up with some of the issues with AI in terms of accuracy, contextual deafness. Um, I'm wondering if, if you had any experience with any of that, did it Overcorrect did it, did it correct someone's speech or grammar? Did you find that that happened as well?

23:41 Zack Ellis

That's a good question. I actually don't know the answer to that. Um, probably the audio transcription center folks would have a stronger sense of that. I don't know, Caitlin, if you remember hearing any feedback from them specifically on if that was something they, they ended up having to, to correct.

23:59 Caitlin Biggers

Not, not generally, I don't think it was a huge problem, but I did come across one just a couple of days ago. That was not, um, exactly this case, but, uh, it was a Sally field recording that had been transcribed by a AI and it, she kept saying, oh God, Sal, like referring to herself. And the transcript was always correcting it to God's hell. Like God's hell, right? Oh, God. Sal. Like it was like very, very similar. So there were cases like that where it's like, I can see why the, um, transcription tool is making the choice that God's hell makes more sense than, oh God. Sal. Um, but yeah, that's the only thing that I can think of really. It didn't seem to be a big problem for our project.

24:40 Zack Ellis

Yeah.

24:41 Jenni Matz

So I. Oh, go ahead.

24:43 Zack Ellis

I was saying one thing to comment on there too is that even with, you know, more improved speech to text models today, I would still say that the thing that we notice that AI gets wrong the most are the things that historians and researchers probably care about the most. Proper nouns, especially like in that example. But but in general, whether it's a person's name, whether it's a place, an acronym of an organization. It's it is oftentimes those types of things that AI gets wrong the most. And there are things that that can be done to give like a, a list of vocabulary terms. And

25:25 Jenni Matz

And that was my question.

25:26 Zack Ellis

Yeah. Yeah. But, but I think vocabulary. Yeah. But, but this is where I, I think, you know, for folks like us and, and other, um, you know, technology providers, um, that are making tools for archives, libraries, museums, um, universities to make sense of, um, human testimony and other types of sources. A lot of the innovation I think is, is at the interface level. It is how do we help people to do this work and to work with the data in more intuitive ways. And I think, for example, one of the areas that can be most helpful, for example, for correcting a transcript, is actually being able to do named entity recognition together with transcription, because then you can look for saying, you know, what are all of the places where a person is referenced and then be able to quickly go through and spot check where a person is referenced or a proper noun in general. So there are combinations of tools and interfaces to help better authenticate AI generated content. And there's cases where you may just want to do it from scratch to begin with. But, but I think that that is, is a place to, um, to help is in the interfaces themselves.

26:55 Jenni Matz

So before I turn to some of the, the peril part of the conversation in terms of unintended enhancements, things that you didn't realize the tech could do or provide, like, you know, finding themes through natural language search. Um, you know, this named entity recognition you're talking about was I guess this is for Caitlin. What were some of the unintended bonuses of using AI for this project?

27:23 Caitlin Biggers

I mean, I think the, the main benefit of using AI for us was really just the, the time it took and it, it needed to be done because of, um, you know, grant terms are always fixed and you, you have a limited amount of money that you can spend on it. So that was the main thing. Um, but I did actually find sometimes that using AI, like it was better, um, at, for poor quality recordings, like you don't always get lucky, but there were some things that I could not really make out with my own ears and having AI suggest what it might be saying was helpful. So that to me was kind of like an unintended or unexpected benefit of like, if someone gives you a guess at what it might be, it's much easier to hear something than trying to pick it out yourself. And I didn't do all of the authentication myself, but that was one I did very little of it myself. But the things that I, um, heard there, I found it was interesting that it's nice to, to start with something as opposed to trying to hear every word yourself.

28:23 Jenni Matz

Um, Andrew, great question in the chat, we're going to get to informed consent, but before we get there, I just sort of want to level set on L. Some people might not know what that means. It's very scary to many of us, um, who are archivists who are worried about these large language models training their data sets based on anything we put online. Right? So you mentioned that this whole system is stored off. Is it offline? Can you explain more about that and why you did that and what that means? Can it not be scraped by?

29:00 Zack Ellis

LM sure, so a couple parts to that question in terms of how we store data. We host on AWS, the recordings themselves, um, in like industry standard, uh, sort of cloud storage, um, capacities in terms of the abilities to have that type of search that I showed that was more like thematic. It wasn't the traditional keyword search. Um, for that, we used one of the industry's leading, um, vector databases that's open source, uh, called v eight. And um, this, we also host on our AWS infrastructure. So as much as possible, we are moving towards a model where the data, like if you trust us for hosting the recording itself, it's the same level of trust to be able to use the AI capabilities, that do allow for richer discovery, um, and generating indexes and named entities and that sort of thing. Um, where we can say the data doesn't leave our AWS infrastructure and it's not going to train third party AI models and nor our own. Um, so that is, um, uh, what we used for, uh, for this project. And actually what we standardize on today is we V8 for that more, um, AI driven search types of capabilities.

30:39 Jenni Matz

Okay. And Eli had a question in the chat about does TheirStory integrate with dams or are we stuck on your platform?

30:46 Zack Ellis

That's a great question. Uh, so interoperability is, uh, key is paramount. Um, so yeah, uh, two of our core principles are never lose customer data and make sure that the data is interoperable in standard formats. So we integrate with a number of platforms today from um, systems like collective access to, uh, permanent.org to aviary, um, where you can just be able to sync your content between platforms. Uh, collective access has one, um, uh, through the nine over 11 museum where they can actually pull the data from collective access or from TheirStory to collective access. Uh, and then you can also always download, uh, the data through the interface. So whether it's the metadata as a CSV or, uh, video as an MP4 or audio as a wave. Um, and then the transcripts and indexes themselves are all exportable. Um, the, uh, the transcript you can export as a text file, a word document as SRT or for closed captions for accessibility purposes. A number of other formats that you can get it out as. And then the same with the index. You can export that as a text file and markdown again. So very standard CSV, XML or for more interactive formats. And then also all of the data on TheirStory is accessible via our API as well, which is all based around the permissions, um, that people have within, within the system. So through the API, you can only access your own data. Um, but also you can be able to build your own custom interfaces or get things out more easily in bulk through the API as well.

32:38 Jenni Matz

Okay. And I think we can probably open it up if people want to unmute and ask questions directly. But, um, just going back to Andrew's question about informed consent, I know we chatted in prep for this a bit. Caitlyn, you weren't really dealing with oral histories per se, where this would be relevant, but I know that that's a concern for my organization at the Television Academy, where we do have oral histories. And, you know, those of you from the Oral History Committee and Amea know all about informed consent, right. And, um, really communicating to the subject what the parameters of use are going to be with the project. Um, so for us, it's becoming a concern and people are starting to ask about it when they sign our appearance releases. How will this be used? Will it be integrating with any AI? Um, now we don't make our transcripts publicly available in part for some of those reasons dealing with privacy and accuracy. Um, since we deal with mostly a lot of celebrity interviews, um, and don't need that, I get a headache. Um, but I am curious if there are people on this call today who are using AI and have had that issue come up and how you're addressing it. Um, we have modified our release a bit, um, to include some of these potential future uses. Um, but you know, we currently only use human transcription. Um, but it is something we're considering, uh, moving forward because that is not feasible economically to continue forever. Uh, we're going to have to turn to AI as many of us will. So, um, does Andrew, did you have more to that question? I don't know if you're still there. Um.

34:26 Zack Ellis

And Jenni maybe. Yeah, while it wasn't oral history, I think there were some interesting legal considerations, um, for this project to maybe Caitlyn, if you wanted to touch on what those were for, uh, for you in this case.

34:42 Caitlin Biggers

Sure. Yeah. Um, you know, like Jenni and Zack said, the informed. Unfortunately, we couldn't inform T. S Eliot and of what our intended use, because he's been dead for a long time, so it wasn't relevant there. Um, but we did have concerns about making sure that we were using the recordings, um, in a way that we had the legal right to, or at the very least since we can't, um, sometimes archival outreach when you're talking about estates, uh, where you're dealing with many, many generations of family members, it can be very difficult to track down the correct, um, copyright permissions. So we had internal conversations kind of about how to deal with this to do the absolute best that we can in, um, making sure that we had either releases that covered this usage or getting in touch with the estates where possible. But what we ended up, um, wanting to make sure was a part of our site was a very clear takedown. Notice that if you know, if you come across something on our site and you think that you own the right to it and you don't want it up there. Get in touch with us. We will take it down immediately while we investigate. But frankly, if you don't own the copyright to it, but you know that this person doesn't want it online, put us in touch with them. Like we're basically just desperate for people to let us know if we're using something in a way that they're not comfortable with. So we wanted to design a way that is as easy as possible to get in contact with us. So that was part of the conversation from the start. Um, because, you know, I think it would be a shame to exclude all content from things that people couldn't have fathomed. Like, it doesn't make sense to me to not put something online because the person died before the internet was invented. You know, we wouldn't have the Odyssey. You know, things like that. It would only be on a, you know, really old scroll. So I think it's important to consider that like, it's not perfect. We can't ask every person, um, whether this is something that they're comfortable with, but we need to just leave space to allow people to say that they're not comfortable with it. Um, and we actually, we haven't really had many people come to us and say, this isn't okay. Um, maybe, you know, but we're still, of course, open to that possibility. But there was a lot of conversations about, yeah, what, what do we do here? And, and we also wanted to make sure that the AI model was not using this data to train, uh, or to train AI models, because that seems to be a whole different category of things that get people upset. So we just felt it was better to just assume that people aren't comfortable with that. Um, and that was just kind of a judgment call on our side. Yeah.

37:23 Jenni Matz

Yeah. Um, and then, you know, on the, it's always the push and pull, right of, we want to provide the best access to our collections. We want them to be totally discoverable and searchable. But, you know, there's always the other hammer of what you're putting out there. And along those lines, I don't know if this came up with this project per se, but in terms of sensitive material, people talking about Murder, committing crimes, defamation, um or personal identifying information that, you know, we, we started all of our interviews in the early days with what was your mother's maiden name? What street did you grow up on? Um, which were now scrubbing. So, um, I'm just curious if anything like that, if can AI be used to scan for that material? Um, and have you had any, um, you know, sort of red flags come up with sensitive information and data with this project.

38:24 Zack Ellis

Caitlin, do you want to start?

38:26 Caitlin Biggers

Yeah, just for us, it didn't come up a ton because it's 99% event recordings on a public stage. And people usually luckily don't say, hi, my name is this and this is my social security number. Um, so we kind of felt like if they were willing to say it in a public forum, generally they were okay with it. Um, but I mean, people do talk about, you know, like I live in the village or something like that, that maybe they wouldn't, you know, be comfortable with now. Um, but I'm not familiar with whether TheirStory can scan for that or not. So. Zack.

38:55 Zack Ellis

Yeah. This is something that has come up recently and I think is one of the interesting uses of AI. And again, it, it more and more points to the need to be able to if you are going to take advantage of AI, the need to be able to use it privately and securely. And it's one of the reasons, um, because there is a lot that it can do, but to make sure that it's not training third party models, um, you know, for, for this project, and I'll come back specifically to that sort of risk analysis or PII, that, that sort of thing in particular. But, you know, for this project, uh, you know, for example, we used the GPT API for, um, the large language model aspect for indexing and in two OpenAI's policy. They do not. They state that they do not train on data that goes through the API. If you have a free account with ChatGPT, anything you say in there is being used to train the model, but if you're using the API, it doesn't. At the same time, people still have questions and concerns about what's happening. And so, you know, we're even moving to a model of allowing or using Amazon bedrock. Um, which means again, we can host, uh, multiple different large language models and be able to switch between models or open source large language models, but it all stays on our AWS server. So again, it's the same level of protection if you trust us for hosting the recording. Uh, and what many people are already used to with AWS hosting, um, media. Uh, then your data stays private, doesn't leave the servers. Uh, and it's not being used to, to train third party, um, models and it's not even going to them. Um, so that's, that's a place that, that we're going and that allows you to then use it for things like what we were just talking about. So scrubbing for PII is definitely something that, um, uh, I know others are starting to do. This is one of the key areas that we are expanding in, um, is the ability, uh, to really, um, customize what the AI is looking for. So today in TheirStory, you actually already have the ability, if you want to, um, say generate an index, you can customize the prompt however you want and you can apply that across, you know, any number of, of your recordings. Um, but even more gradually than that, what we're working on right now is, um, you know, a lot of people may use AI for looking for thematic elements, but even beyond specific themes, or if you have a controlled vocabulary, you're looking to extract those across recordings. Um, is you could define your own parameters here for what you would want it to look for. And instead of a theme that you know, is leaving home or whatever, um, it would be this type of risk and you can define what you want the large language model to, to look for. Um, and it could, for example, you could say, here are examples of PII. Look for this and surface the exact moments in the recordings where, where this takes place. Uh, and so this is the, the next thing that we're developing, it's not available today, but this is, I think part of the direction is giving institutions control over the ways in which they fine tune, if you will, um, the prompts and have many different types of prompts that they can leverage on, on the collection.

42:44 Jenni Matz

Yeah. Um, actually, if you want to go back and show an example of how you edit in the platform, Andrew just came up. Andrew. All the questions today, I love it. Um, Andrew asked, are there options for community input in correcting or remediating transcripts if something is caught?

43:00 Zack Ellis

Yeah, yeah. So in the project with 92nd Street Y, it was limited to, um, the, uh, you know, audio transcription center contractors, you know, authenticators to be editing transcripts. Um, but obviously a lot of times if it's, uh, you know, a new oral history project, you're interviewing someone and then you may want to be able to allow them to create the metadata or, you know, uh, review the transcript. So yeah. So there are, um, uh, while that didn't take place in the 92nd street y project, just based on the nature of it, uh, there is the ability in, in TheirStory. If you are um correcting a transcript, um, first it takes place with a logged in account, but you do have the ability to share the transcript out, say with a narrator, uh, in either in like a view only, uh, format where they can't actually make edits to the transcript. Or if you, um, choose the full access link, then they can actually go in and make edits themselves and they don't need an account. Uh, in this case, so you can just have a URL that you share with, say directly with the narrator. They can come in, they can spot check, they can make edits, you know, communicate that back. Um, uh, ultimately. And then you always have the ability, uh, to, uh, to reset that link so that the, the link isn't then always open should it accidentally get shared or anything like that. You can always make sure that you can generate a new link and lock down any, uh, any old links.

44:30 Jenni Matz

Yeah. And I know we've had experience with our own projects with just, you know, doing a sort of search all for a name and replacing all of it if it was misspelled one time. Yeah. It's just like a easy to do. Um, I did want to talk about, so a lot of what we are being asked for now as an example, um, are ways to monetize our collections. I'm sure there are people out there, um, who have a similar, you know, raise our own money to be able to sustain our archives. Um, so we're asked to often find like the gems in our collection, right? What are the best quotes, you know, and our push pull with that. And, um, it's, it's, you know, I see it, the potential in TheirStory, as with the project you're doing with us now, Zack, to identify those themes and to pull out those quotes and actually edit them and make, you know, make them accessible that way. But is there a worry about decontextualizing? Um, the context, the greater context is there, um, you know, are there any tools to maintain, um, any kind of contextualization for the interview so that if you pull these short clips and quotes, we can still understand the larger axis. I'm just wondering from a design level, if that's something you have been working on.

45:53 Zack Ellis

Yeah, maintaining context is definitely a huge part. And what gets lost when, you know, a clip gets shared to social media, uh, so often is how do we get back to the source and understand if it's in or out of context? And I know Caitlin, we were talking about that, um, as well. Um, yeah, so there's a few considerations that, that we have, uh, for this. Um, let me go back. Uh, so one is what happens in the platform. And then there's also what, what happens out of the platform. So, uh, in, in the platform. And, and this is, um, just a demo instance. This isn't anyone's real, real data. Uh, in the platform, there is the ability to, um, you know, create clips, uh, and then use those out of the system, but always making sure at the very least in the system, it's connected back. So I'm just going to search for a term here, let's say immigrant. Um, and that'll search both the index here. Uh, it'll also search, um, down to the transcript. And then, uh, if I wanted to use any of this here, um, this happens to be a very moving, uh, excerpt. Um, then I'll just highlight that part of the transcript and say create clip and I'll give this a title. We're all immigrants at this time. Give it a keyword here. Let's say demo. Uh, and then you can see I demoed this earlier. Um, and then that automatically generates a video or audio, um, file that you can extract and use how you want now in the system. Let's say there's someone on the marketing team that wants to come in and use any of these clips and use them appropriately, um, or for a podcast or whatever that might be. If you go to the clips library, then you can find all those clips and let me search for a keyword. We can see all the clips I've created for demo purposes. Um, and if I click on one of those clips, I can always say go to interview. And that will then take me right back to that exact moment in the recording so I can get to the context really easily. Um, now sharing outside of the, of the system, um, part of what um, I think about is if someone does download the clip itself, you know, there's what metadata can go along with it, but there's also, um, if someone shares, I'll just use an example here. Um, uh, a newer portal that's been created. If, um, if someone shares something, uh, external, um, do we have a way, if they're sharing an excerpt to make it easy for someone to find that moment, but it, it maintains the context. So one of the things that we do is, uh, based on the URL itself, if I do something like start equals 200 and, and equals 300. Um, so we can allow sharing that is of a specific moment. So we saw that when I updated when I just hit enter there, refreshed the page. Uh, it went right to that moment. So we actually haven't enabled it yet here, but what we'll be doing next is if I say, find a piece of media that I want to share or a moment, um, is if I then share that out, that it'll share with the. Because we know the start and end time of every word, it can share that exact, um, uh, moment to encourage people to share in this way where it can still be the exactness of the moment, but bring people back to, to context. That's one way that, that we're thinking about it. But there is always that challenge of once you download the clip itself, and if you just post that, how do you maintain the context? And that, that I think is something that we still need to think about, um, more.

49:59 Jenni Matz

So, um, we just are about ten minutes left today. So if anyone does have questions, I encourage you to, I mean, feel free to unmute at this point, but, um, I, I, I was reviewing, um, great Mary Larson article in the oral history review, um, about AI and one of the suggestions for best practices was to consider bot management platform. So if you have this site that TheirStory creates you or you create yourself that is public. Are there ways that you can say go bots, go away? LMS not welcome here. Um, what are what are some other, um, you know, tools that we can use to protect?

50:44 Zack Ellis

Yeah.

50:44 Jenni Matz

Our IP that we don't want scraped.

50:47 Zack Ellis

Yeah, yeah. So I think this is where, um, choosing the right tools is important. Um, one so the platform that we use for streaming video is called mux.com. Um, and they are fantastic and have a lot of different, um, capabilities. So I would definitely recommend for any institution here to look at MUX even just independently of TheirStory. Um, they make like state of the art, very easy to use from a software developer standpoint. Um, streaming video components. And part of what they allow for are things like, um, making it hard for us automate like automated screen capture of a video or, um, screenshots, things like that. Um, they're really robust. So I think they're going to continue to innovate in this space of what can an agent do and what can't it do? Um, with video content. Now, another thing that we consider, um, when creating these types of portals, you know, this is something you can, you know, just do through, through our system, um, is whether or not the transcript itself, um, is, uh, is exposed. Oh, I might not have been sharing my, my screen there.

52:07 SPEAKER_S4

Um, you are.

52:08 Zack Ellis

I am okay. Um, yeah. Whether or not the transcript is exposed. So this is something where our default has been, uh, and I, um, shared their purposely, but our default has been to not expose the transcript, and it gets dynamically generated when you load the page. Um. Although I can, but there is this tension between, you know, if someone goes to ChatGPT or to Claude or Gemini and searches for something that something in your collection might surface. Well, do you not want your primary sources to be surfaced there? So, um, and do you not want the, you know, exact moment that's relevant to be surfaced there and that, you know, that does take then exposing the, the transcript likely, you know, this is something where we still need to explore more. But so there is that tension. I think our default is to try to give control to the institution who's using the platform. And so as much as possible, like when you create a research portal, you can choose to make it private and password protected. You can choose. Can people download or not? And I think one of the next things that we'll need to build into the system is the option of, do we want other agents to be able to read us the transcript data and be able to pull that into the model so that when someone searches something relevant, that it brings the that exact moment. I know Caitlin, you were saying before, well, maybe it makes sense to us to surface, uh, tags that are more at like the collection level. And so maybe you get the interview, but you don't get, you know, that exact moment and it's not training on the transcript. So these are things that we need to wrestle with as a provider to folks in this space is to just keep having these conversations about what options do you want? I don't know, Caitlin, if, if I paraphrased what you were thinking well or missed, or any color you want to add to that. No,

54:19 Caitlin Biggers

No, totally. You got it. I mean, I think that, yeah, it's like when we have these really robust tools for finding things, you do want to be included in the discovery, I think, but there is a risk, um, of, yeah, we want to keep it in context and directing back to us and not just get broken up across the internet where suddenly a quote is plucked out and attributed to someone else because it's just been like AI to death. Um, but I think it would be a kind of a tragedy if it was just totally hidden from these tools where people are increasingly searching for things. So yeah, it's something to think about. Um, there's pros and cons.

54:56 Jenni Matz

Our perpetual saga of. Um, well, we're almost at the, the, the hour mark. Um, oh, here's another question from our buddy Andrew. Okay, I'll read it out. Are there security measures put in place if a portion of the transcript or video is shared from TheirStory after download independently to LMS. Specifically, is there a way for the LM to recognize, oh, we can't view this, so we can't answer it for you. Um.

55:27 Zack Ellis

Yeah. Interesting. I'm, I think I'm understanding the question. Um, but Andrew, um, please, as I answer in the chat or come off mute, let me know if I'm misinterpreting. So, um, in terms of the, the system that, that we have here, um, you know, there is the ability within the system itself to, um, you know, ask questions like you would in ChatGPT, but it's localized to this specific collection. Now this was something that we actually decided or we had originally built, um, in the 92nd street y research portal and Caitlin had said, um, hey, this is amazing, uh, for being able to explore the collection, but, um, this is largely, uh, you know, creatives, writers, um, and they're reading their works and this can look or actually be, you know, a large language model interpreting their works. And so as amazing as this would be for like internal use, we don't want this public, this public capability. Um, because it could be seen as interpret interpreting, um, writers works. And that would be an added layer of, of permissions and having to go back to each person say, are you okay with this? And so we actually turned off this capability. And again, it's another area where we give choice. When you're setting up this infrastructure, you can literally with a flip of a switch, turn on or off this capability. Um, but within, within the, TheirStory context, when you use this capability, if I ask for something, you know, uh, this happens to be the showing it here because it's not on in the 90s. But you know, what does this collection say about the 92 and why? Um, you know, poetry center, uh, and, uh, if, you know, my live demonstration goes as I anticipate, uh, into your question, it should say, I don't know. Um, like that's not, um, a part of this. Um, yeah. And so it says, you know, none of the interviews, chapter summaries, you know, talk about this. And so we, we have it, um, working in that way where it's going to answer specifically about what's in the collection and not outside of it. Now, I think there's a second part to your question, which is. Data that is shared outside of the TheirStory platform. And let's say that gets into a large language model and you ask about it. Is there some way to mediate that. And I don't know the the answer to to that yet. I think that's an area for us to keep exploring is is there something we can put into the metadata or something along those lines, uh, that if a clip is downloaded that some of those constraints can still be there? I don't know the the answer to that, but it definitely feels like a harder problem.

58:30 Jenni Matz

Mhm.

58:31 Zack Ellis

What can you control on platform versus versus off.

58:35 Caitlin Biggers

Like a digital watermark of sorts?

58:37 Zack Ellis

Yeah, yeah, I, I do know that, um, there is a emerging standard called Ctpa, which is a lot about, um, content provenance, um, and authenticity and detecting if something's AI generated or edited. Um, so that is definitely an area for further exploration for us. Um, as well, our emerging standards like Ctpa that are tackling these questions of provenance and authenticity and what can I do and not do with content or even just detection of what's been done.

59:12 Jenni Matz

I'm going to wrap up quickly. This has been amazing. Thank you, Caitlin and Zack, so much for sharing all of this. Um, I think there's a lot more of this conversation to be had, so maybe we will reconvene with a follow up, um, panel at some point. Um, please do contact me. I, I, I'm curious how many of you are coming from the copyright side? How many are coming from the oral history side? Let us know what other questions or details that we didn't get to today. You'd like us to dig in on. And maybe we'll be fortunate enough to have Zack and Caitlin back. Um, maybe with some other friends too. Um, so thank you, everybody for joining us today. We are going to stop the recording. Um, and if, uh, how do I stop the recording? You started it.

59:58 Zack Ellis

I will stop the recording. Thank you.

1:00:01 Jenni Matz

Uh, so, uh, definitely, um, I think.

About TheirStory

Ask about the interviews

About TheirStory

Ask about the interviews