PhD A: Eh, we should be going. Professor B: So ne next week we'll have, uh, both Birger {pause} and, uh, Mike {disfmarker} Michael {disfmarker} Michael Kleinschmidt and Birger Kollmeier will join us. PhD D: Uh - huh. Professor B: Um, and you're {disfmarker} {vocalsound} you're probably gonna go up in a couple {disfmarker} three weeks or so? When d when are you thinking of going up to, uh, OGI? PhD D: Yeah, like, uh, not next week but maybe the week after. Professor B: OK. Good. So at least we'll have one meeting with {vocalsound} yo with you still around, and {disfmarker} and {disfmarker} PhD D: Uh - huh. Professor B: That's good. PhD D: Um, Yeah. Well, {vocalsound} maybe we can start with this. Mmm. Professor B: All today, huh? PhD D: Yeah. Professor B: Oh. PhD D: Um. Yeah. So there was this conference call this morning, um, and the only topic on the agenda was just to discuss a and to come at {disfmarker} uh, to get a decision about this latency problem. Professor B: No, this {disfmarker} I'm sorry, this is a conference call between different Aurora people or just {disfmarker}? PhD D: Uh, yeah. It's the conference call between the Aurora, {vocalsound} uh, group. Professor B: It's the main conference call. OK. PhD D: Uh, yeah. There were like two hours of {pause} discussions, and then suddenly, {vocalsound} uh, people were tired, I guess, and they decided on {nonvocalsound} a number, two hundred and twenty, um, included e including everything. Uh, it means that it's like eighty milliseconds {pause} less than before. Professor B: And what are we sitting at currently? PhD D: Um. Professor B: Yeah. PhD D: So, currently d uh, we have system that has two hundred and thirty. So, that's fine. Professor B: Two thirty. PhD D: Yeah. So that's the system that's described on the second point of {pause} this {vocalsound} document. Professor B: So it's {disfmarker} we have to reduce it by ten milliseconds somehow. PhD D: Yeah. But that's {disfmarker} Yeah. That's not a problem, I {disfmarker} I guess. Professor B: OK. W It's {disfmarker} it's p d primary {disfmarker} primarily determined by the VAD at this point, PhD D: Um. Professor B: right? PhD D: Yeah. Professor B: S so we can make the VAD a little shorter. PhD D: Yeah. At this point, yeah. Professor B: That's {disfmarker} PhD D: Yeah, uh - huh. Professor B: Yeah. We probably should do that pretty soon so that we don't get used to it being a certain way. PhD D: Uh - huh. Professor B: Yeah. PhD D: Um. Professor B: Was Hari on the {disfmarker} on the phone? PhD D: Yeah, sure. Professor B: OK. PhD D: Well, it was mainly a discussion {vocalsound} between Hari and {vocalsound} David, Professor B: Hmm. PhD D: who was like {disfmarker} Professor B: Yeah. PhD D: Uh, Professor B: OK. PhD D: mmm {disfmarker} Uh, yeah. So, the second thing is the system that we have currently. Oh, yes. We have, like, a system that gives sixty - two percent improvement, but {vocalsound} if you want to stick to the {disfmarker} {vocalsound} this latency {disfmarker} Well, it has a latency of two thirty, but {vocalsound} if you want also to stick to the number {vocalsound} of features that {disfmarker} limit it to sixty, {vocalsound} then we go a little bit down but it's still sixty - one percent. Uh, and if we drop the tandem network, then we have fifty - seven percent. Professor B: Uh, but th the two th two thirty includes the tandem network? PhD D: Yeah. Professor B: OK. And i is the tandem network, uh, small enough that it will fit on the terminal size in terms of {disfmarker}? PhD D: Uh, no, I don't think so. Professor B: No. PhD D: No. Professor B: OK. PhD D: It's still {disfmarker} in terms of computation, if we use, like, their way of computing the {disfmarker} the maps {disfmarker} the {disfmarker} the MIPs, {vocalsound} I think it fits, Professor B: Mm - hmm. Mm - hmm. PhD D: but it's, uh, m mainly a problem of memory. Professor B: Right. PhD D: Um, and I don't know how much {pause} this can be discussed or not, because it's {disfmarker} it could be in ROM, so it's maybe not that expensive. But {disfmarker} Professor B: Ho - how much memory d? H how many {disfmarker}? PhD D: I d I d uh, I {disfmarker} I don't kn remember exactly, but {disfmarker} {vocalsound} Uh. Yeah, I c I {disfmarker} I have to check that. Professor B: Yeah. I'd like to {pause} see that, cuz maybe I could think a little bit about it, cuz we {vocalsound} maybe we could make it a little smaller or {disfmarker} I mean, it'd be {disfmarker} it'd be neat if we could fit it all. PhD D: Uh - huh. Professor B: Uh, I'd like to see how far off we are. PhD D: Mm - hmm. Professor B: But I guess it's still within their rules to have {disfmarker} have it on the, uh, t uh, server side. Right? PhD D: Yeah. Yeah. Professor B: OK. PhD D: Mmm. Professor B: And this is still {disfmarker}? Uh, well, y you're saying here. I c I should just let you go on. PhD D: Yeah, there were small tricks to make this tandem network work. Uh, {vocalsound} mmm, and one of the trick was to, {vocalsound} um, use {vocalsound} some kind of hierarchical structure where {pause} the silence probability is not computed by {pause} the final tandem network but by the VAD network. Um, so apparently it looks better when, {vocalsound} uh, we use the silence probability from the VAD network Professor B: Huh. PhD D: and we re - scale the other probabilities by one minus the silence probability. Um. So it's some kind of hierarchical thing, {vocalsound} uh, that Sunil also tried, um, {vocalsound} {vocalsound} on SPINE and apparently it helps a little bit also. Mmm. And. Yeah, the reason w why {disfmarker} why we did that with the silence probability was that, {vocalsound} um {disfmarker} Professor B: Could {disfmarker}? Uh, uh, I'm {disfmarker} I'm really sorry. Can you repeat what you were saying about the silence probability? PhD D: Mm - hmm. Professor B: I only {disfmarker} My mind was some {disfmarker} PhD D: Yeah. So there is the tandem network that e e e estimates the phone probabilities Professor B: Yeah. Yeah. PhD D: and the silence probabilities also. Professor B: Right. PhD D: And {vocalsound} things get better when, instead of using the silence probability computed by the tandem network, we use the silence probability, uh, given by the VAD network, Professor B: Oh. PhD D: um, Professor B: The VAD network is {disfmarker}? PhD D: Which is smaller, but maybe, um {disfmarker} So we have a network for the VAD which has one hundred hidden units, and the tandem network has five hundred. Um. So it's smaller but th the silence probability {pause} from this network seems, uh, better. Professor B: OK. PhD D: Mmm. Uh. Well, it looks strange, but {disfmarker} Professor B: Yeah. But {disfmarker} PhD D: but it Professor B: OK. PhD D: Maybe it's {disfmarker} has something to do to {vocalsound} the fact that {vocalsound} we don't have infinite training data and {disfmarker} Professor B: We don't? PhD D: Well! And so {disfmarker} Well, things are not optimal Professor B: Yeah. PhD D: and {disfmarker} Mmm {disfmarker} Grad E: Are you {disfmarker} you were going to say why {disfmarker} what made you {disfmarker} wh what led you to do that. PhD D: Yeah. Uh, there was a p {comment} problem that we observed, um, {vocalsound} {vocalsound} that there was {disfmarker} there were, like, many insertions in the {disfmarker} in the system. Professor B: Mm - hmm. PhD D: Mmm. Professor B: Hmm. PhD D: Actually plugging in the tandem network was increasing, I {disfmarker} I {disfmarker} I think, the number of insertions. Professor B: Mm - hmm. PhD D: And, {vocalsound} um {disfmarker} So it looked strange and then just using the {disfmarker} the other silence probability helps. Mmm. Um {disfmarker} Yeah. The next thing we will do is train this tandem on more data. Professor B: So, you know, in a way what it might {disfmarker} i it's {disfmarker} it's a little bit like {vocalsound} combining knowledge sources. PhD D: Um {disfmarker} Professor B: Right? Because {vocalsound} the fact that you have these two nets that are different sizes {pause} means they behave a little differently, PhD D: Mm - hmm. Professor B: they find different {pause} things. And, um, if you have, um {disfmarker} f the distribution that you have from, uh, f speech sounds is w {comment} sort of one source of knowledge. PhD D: Mm - hmm. Professor B: And this is {disfmarker} and rather than just taking one minus that to get the other, which is essentially what's happening, you have this other source of knowledge that you're putting in there. So you make use of both of them {vocalsound} in {disfmarker} in {pause} what you're ending up with. Maybe it's better. PhD D: Yeah. Professor B: Anyway, you can probably justify anything if what's use PhD D: Yeah. Professor B: Yeah. PhD D: And {disfmarker} and the features are different also. I mean, the VAD doesn't use the same features there are. Professor B: Mm - hmm. Grad E: Hmm. Professor B: Oh! PhD D: Um {disfmarker} Professor B: That might be the key, actually. PhD D: Mm - hmm. Professor B: Cuz you were really thinking about speech versus nonspeech for that. PhD D: Mm - hmm. Professor B: That's a good point. PhD D: Mmm. Uh. Well, there are other things that {vocalsound} we should do but, {vocalsound} um, {vocalsound} it requires time and {disfmarker} {vocalsound} We have ideas, like {disfmarker} so, these things are like hav having a better VAD. Uh, we have some ideas about that. It would {disfmarker} {vocalsound} probably implies working a little bit on features that are more {vocalsound} suited to a voice activity detection. Professor B: Mm - hmm. PhD D: Working on the second stream. Of course we have ideas on this also, but {disfmarker} {vocalsound} w we need to try different things and {disfmarker} Uh, but their noise estimation, um {disfmarker} {vocalsound} uh {disfmarker} Professor B: I mean, back on the second stream, I mean, that's something we've talked about for a while. I mean, I think {nonvocalsound} that's certainly a high hope. PhD D: Yeah. {vocalsound} Mmm. Professor B: Um, so we have this {disfmarker} this default idea about just using some sort of purely spectral thing? PhD D: Uh, yeah. Professor B: for a second stream? PhD D: But, um, we {disfmarker} we did a first try with this, and it {disfmarker} it {vocalsound} clearly hurts. Professor B: But, uh, how was the stream combined? PhD D: Uh. {vocalsound} It was c it was just combined, um, by the acoustic model. So there was, no neural network for the moment. Professor B: Right. So, I mean, if you just had a second stream that was just spectral and had another neural net and combined there, that {disfmarker} that, uh, {vocalsound} might be good. PhD D: Mm - hmm. Yeah. Mm - hmm. Mm - hmm. Mmm. Yeah. Um {disfmarker} Yeah, and the other thing, that noise estimation and th um, maybe try to train {disfmarker} uh, the training data for the t tandem network, right now, is like {disfmarker} i is using the noises from the Aurora task and {vocalsound} {vocalsound} I think that people might, {vocalsound} um, try to argue about that because {vocalsound} then in some cases we have the same noises in {disfmarker} for training the network {pause} than the noises that are used for testing, Professor B: Right. PhD D: and {disfmarker} So we have t n uh, to try to get rid of these {disfmarker} {vocalsound} this problem. Professor B: Yeah. Maybe you just put in some other noise, something that's different. PhD D: Mm - hmm. {vocalsound} Yeah. Professor B: I mean, it {disfmarker} it's probably helpful to have {disfmarker} have a little noise there. But it may be something else PhD D: Uh - huh. Professor B: th at least you could say it was. PhD D: Yeah. Professor B: And then {disfmarker} if it doesn't hurt too much, though. PhD D: Uh - huh. Professor B: Yeah. That's a good idea. PhD D: Um. Yeah. The last thing is that I think we are getting close to human performance. Well, that's something I would like to investigate further, but, um, I did, like, um {disfmarker} I did, uh, listen to the m most noisy utterances of the SpeechDat - Car Italian and tried to transcribe them. And, um {disfmarker} Professor B: So this is a particular human. This is {disfmarker} this i this is Stephane. PhD D: Yeah. So that's {disfmarker} that's {disfmarker} Grad E: St - Stephane. Professor B: Yeah. PhD D: that's the {disfmarker} the flaw of the experiment. This is just {disfmarker} i j {comment} {vocalsound} {vocalsound} it's just one subject, Professor B: Yeah. Grad E: Getting close. PhD D: but {disfmarker} but still, uh, {vocalsound} what happens is {disfmarker} is that, {vocalsound} uh, the digit error rate on this is around one percent, Professor B: Yeah. PhD D: while our system is currently at seven percent. Um, but what happens also is that if I listen to the, um {disfmarker} {nonvocalsound} a re - synthesized version of the speech and {pause} I re - synthesized this using a white noise that's filtered by a LPC, uh, filter {disfmarker} Professor B: Yeah. PhD D: Um, well, you can argue, that, uh {disfmarker} that this is not speech, Professor B: Yeah. PhD D: so the ear is not trained to recognize this. But s actually it sound like {pause} whispering, so we are {disfmarker} Professor B: Well, I mean, it's {disfmarker} PhD D: eh {disfmarker} Professor B: There's two problems there. I mean {disfmarker} I mean, so {disfmarker} so the first is {vocalsound} that by doing LPC - twelve with synthesized speech w like you're saying, uh, it's {disfmarker} {vocalsound} i i you're {disfmarker} you're adding other degradation. PhD D: Uh - huh. Professor B: Right? So it's not just the noise but you're adding in fact some degradation because it's only an approximation. Um, and the second thing is {disfmarker} which is m maybe more interesting {disfmarker} is that, um, {comment} {vocalsound} if you do it with whispered speech, you get this number. What if you had {pause} done analysis {comment} re - synthesis and taken the pitch as well? Alright? So now you put the pitch in. PhD D: Uh - huh. Professor B: What would the percentage be then? PhD D: Um {disfmarker} Professor B: See, that's the question. So, you see, if it's {disfmarker} if it's {disfmarker} if it's, uh {disfmarker} Let's say it's {pause} back down to one percent again. PhD D: Uh - huh. Professor B: That would say at least for people, having the pitch is really, really important, which would be interesting in itself. Um, PhD D: Uh, yeah. But {disfmarker} Professor B: if i on the other hand, if it stayed up {pause} near five percent, {vocalsound} then I'd say" boy, LPC n twelve is pretty crummy" . You know? PhD D: Uh - huh. Professor B: So I I I'm not sure {disfmarker} I'm not sure how we can conclude from this anything about {disfmarker} that our system is close to {vocalsound} the human performance. PhD D: Ye Yeah. Well, the point is that eh l ey {disfmarker} the point is that, um, {vocalsound} what I {disfmarker} what I listened to when I re - synthesized the LP - the LPC - twelve {pause} spectrum {vocalsound} is in a way what the system, uh, is hearing, cuz @ @ {disfmarker} all the {disfmarker} all the, um, excitation {disfmarker} all the {disfmarker} well, the excitation is {disfmarker} is not taken into account. That's what we do with our system. And Professor B: Well, you're not doing the LPC {disfmarker} PhD D: in this case {disfmarker} Professor B: I mean, so {disfmarker} so what if you did a {disfmarker} PhD D: Well, it's not LPC, sure, Professor B: What if you did LPC - twenty? PhD D: but {disfmarker} LPC {disfmarker}? Professor B: Twenty. Right? I mean, th the thing is LPC is not a {disfmarker} a really great representation of speech. PhD D: Mm - hmm. Mm - hmm. Professor B: So, all I'm saying is that you have in addition to the w the, uh, removal of pitch, {vocalsound} you also are doing, uh, a particular parameterization, PhD D: Mm - hmm. Professor B: which, um, uh {disfmarker} PhD D: Mmm. Professor B: Uh, so, let's see, how would you do {disfmarker}? So, fo PhD D: But that's {disfmarker} that's what we do with our systems. And {disfmarker} Professor B: No. Actually, we d we {disfmarker} we don't, because we do {disfmarker} we do, uh, {vocalsound} uh, mel filter bank, for instance. Right? PhD D: Yeah, but is it that {disfmarker} is it that different, I mean? Professor B: Um, {vocalsound} I don't know what mel, {pause} uh, based synthesis would sound like, PhD D: I Professor B: but certainly the spectra are quite different. PhD D: Mm - hmm. PhD A: Couldn't you t couldn't you, um, test the human performance on just the original {pause} audio? PhD D: Mm - hmm. This is the one percent number. Professor B: Yeah, it's one percent. He's trying to remove the pitch information PhD D: Mm - hmm. PhD A: Oh, oh. OK, PhD D: Mm - hmm. PhD A: I see. Professor B: and make it closer to what {disfmarker} to what we're seeing as the feature vectors. PhD A: OK. So, y uh, your performance was one percent, and then when you re - synthesize with LPC - twelve it went to five. PhD D: Uh - huh. Yeah. PhD A: OK. Professor B: I mean {disfmarker} We were {disfmarker} we were j It {disfmarker} it {disfmarker} it's a little bit still apples and oranges because we are choosing these features in order to be the best for recognition. PhD D: Uh - huh. Professor B: And, um, i if you listen to them they still might not be very {disfmarker} Even if you made something closer to what we're gonna {disfmarker} i it might not sound very good. PhD D: Yeah. Professor B: Uh, and i the degradation from that might {disfmarker} might actually make it even harder, {vocalsound} uh, to understand than the LPC - twelve. So all I'm saying is that the LPC - twelve {vocalsound} puts in {disfmarker} synthesis puts in some degradation that's not what we're used to hearing, PhD D: Uh - huh. Professor B: and is, um {disfmarker} It's not {disfmarker} it's not just a question of how much information is there, as if you will always take maximum {vocalsound} advantage of any information that's presented to you. PhD D: Mm - hmm. Professor B: In fact, you {vocalsound} hear some things better than others. And so it {disfmarker} it isn't {disfmarker} PhD A: But {disfmarker} Professor B: But, {vocalsound} I agree that it says that, uh, the kind of information that we're feeding it is probably, {vocalsound} um, um, a little bit, um, minimal. There's definitely some things that we've thrown away. And that's why I was saying it might be interesting if you {disfmarker} {vocalsound} an interesting test of this would be if you {disfmarker} if you actually put the pitch back in. So, you just extract it from the actual speech and put it back in, and see does that {disfmarker} is that {disfmarker} does that make the difference? If that {disfmarker} if that takes it down to one percent again, {vocalsound} then you'd say" OK, it's {disfmarker} it's in fact having, um, {vocalsound} not just the spectral envelope but also the {disfmarker} also the {disfmarker} the pitch {vocalsound} that, uh, {comment} @ @ {comment} has the information that people can use, anyway." PhD D: Uh - huh. Mmm. PhD A: But from this it's pretty safe to say that the system is with either {vocalsound} two to seven percent away from {pause} the performance of a human. Right? So it's somewhere in that range. Professor B: Well, or it's {disfmarker} it's {disfmarker} PhD A: Two {disfmarker} two to six percent. Professor B: Yeah, so {disfmarker} It's {disfmarker} it's one point four times, uh, to, uh, seven times the error, PhD D: To f seven times, yeah. Professor B: for Stephane. PhD D: Um. Professor B: So, uh {disfmarker} uh, but i I don't know. I do don't wanna take you away from other things. PhD D: But {disfmarker} {comment} but {disfmarker} Professor B: But that's {disfmarker} {vocalsound} that's what {disfmarker} that's the first thing that I would be curious about, is, you know, i i {vocalsound} when you we PhD D: But the signal itself is like a mix of {disfmarker} um, of a {disfmarker} a periodic sound and, {pause} @ @ {comment} uh, unvoiced sound, and the noise Professor B: Mm - hmm. PhD D: which is mostly, {vocalsound} uh, noise. I mean not {pause} periodic. So, {pause} what {disfmarker} what do you mean exactly by putting back the pitch in? Because {disfmarker} PhD A: In the LPC synthesis? I think {disfmarker} Professor B: Yeah. You did LPC re - synthesis {disfmarker} PhD D: I Professor B: L PC re - synthesis. PhD D: Uh - huh. Professor B: So, {vocalsound} uh {disfmarker} and you did it with a noise source, rather than with {disfmarker} with a s periodic source. PhD D: Mm - hmm. Professor B: Right? So if you actually did real re - synthesis like you do in an LPC synthesizer, where it's unvoiced you use noise, where it's voiced you use, {vocalsound} uh, periodic pulses. PhD D: Um. Professor B: Right? PhD D: Yeah, but it's neither {pause} purely voiced or purely unvoiced. Esp - especially because there is noise. Professor B: Well, it might be hard to do it PhD D: So {disfmarker} Professor B: but it but {disfmarker} but the thing is that if you {disfmarker} {vocalsound} um, if you detect that there's periodic {disfmarker} s strong periodic components, then you can use a voiced {disfmarker} voice thing. PhD D: Oh. Uh - huh. Yeah. Professor B: Yeah. I mean, it's probably not worth your time. It's {disfmarker} it's a side thing and {disfmarker} and {disfmarker} and there's a lot to do. PhD D: Uh - huh, yeah. Professor B: But I'm {disfmarker} I'm just saying, at least as a thought experiment, {vocalsound} that's what I would wanna test. PhD D: Mm - hmm. Professor B: Uh, I wan would wanna drive it with a {disfmarker} a {disfmarker} a two - source system rather than a {disfmarker} than a one - source system. PhD D: Mm - hmm. Mm - hmm. Professor B: And then that would tell you whether in fact it's {disfmarker} Cuz we've talked about, like, this harmonic tunneling or {vocalsound} other things that people have done based on pitch, maybe that's really a key element. Maybe {disfmarker} maybe, uh, {vocalsound} uh, without that, it's {disfmarker} it's not possible to do a whole lot better than we're doing. That {disfmarker} that could be. PhD D: Yeah. That's what I was thinking by doing this es experiment, Professor B: Yeah. PhD D: like {disfmarker} Mmm. {vocalsound} Evi Professor B: But, I mean, other than that, I don't think it's {disfmarker} I mean, other than the pitch de information, {vocalsound} it's hard to imagine that there's a whole lot more {vocalsound} in the signal that {disfmarker} that, uh {disfmarker} that we're throwing away that's important. PhD D: Yeah, but {disfmarker} Yeah. {vocalsound} Mm - hmm. Yeah, right. Professor B: Right? I mean, we're using {vocalsound} a fair number of filters in the filter bank and {disfmarker} uh {disfmarker} PhD D: Mm - hmm. Uh, yeah. Professor B: Hmm. Yeah. PhD D: Um. Professor B: Yeah. That look PhD D: Yeah, that's it. Professor B: Yeah. That's {disfmarker} that's {disfmarker} I mean, one {disfmarker} one percent is sort of what I would {disfmarker} I would figure. If somebody was paying really close attention, you might get {disfmarker} I would actually think that if, {vocalsound} you looked at people on various times of the day and different amounts of attention, you might actually get up to three or four percent error on digits. Uh, {vocalsound} uh {disfmarker} PhD D: Mm - hmm. Um. Professor B: So it's {disfmarker} you know, we're not {disfmarker} we're not incredibly far off. On the other hand, with any of these numbers except maybe the one percent, it's st it's not actually usable in a commercial system with a full telephone number or something. PhD D: Uh - huh. Yeah. At these noise levels. Professor B: Yeah. PhD D: Yeah. Mm - hmm. Professor B: Right. PhD D: Well, yeah. These numbers, I mean. Mmm. Professor B: Good. Um, while we're still on Aurora stuff {pause} maybe you can talk a little about the status with the, uh, {vocalsound} Wall Street Journal {vocalsound} things for it. PhD A: So I've, um, downloaded, uh, a couple of things from Mississippi State. Um, one is their {vocalsound} software {disfmarker} their, uh, LVCSR system. Downloaded the latest version of that. Got it compiled and everything. Um, downloaded the scripts. They wrote some scripts that sort of make it easy to run {vocalsound} the system on the Wall Street Journal, uh, data. Um, so I haven't run the scripts yet. Uh, I'm waiting {disfmarker} there was one problem with part of it and I wrote a note to Joe asking him about it. So I'm waiting to hear from him. But, um, I did print something out just to give you an idea about where the system is. Uh, {vocalsound} they {disfmarker} on their web site they, uh, did this little table of where their system performs relative to other systems that have done this {disfmarker} this task. And, um, the Mississippi State system {vocalsound} using a bigram grammar, uh, is at about eight point two percent. Other comparable systems from, uh {disfmarker} {vocalsound} were getting from, uh, like six point nine, six point eight percent. So they're {disfmarker} Professor B: This is on clean test set? PhD A: This is on clean {disfmarker} on clean stuff. Yeah. They {disfmarker} they've started a table {vocalsound} where they're showing their results on various different noise conditions but they {disfmarker} they don't have a whole lot of it filled in and {disfmarker} {vocalsound} and I didn't notice until after I'd printed it out that, um, {vocalsound} they don't say here {pause} what these different testing conditions are. Professor B: PhD A: You actually have to click on it on the web site to see them. So I {disfmarker} I don't know what those {pause} numbers really mean. Professor B: What kind of numbers are they getting on these {disfmarker} on the test conditions? PhD A: Well, see, I was a little confused because on this table, I'm {disfmarker} the they're showing word error rate. But on this one, I {disfmarker} I don't know if these are word error rates because they're really big. So, {vocalsound} under condition one here it's ten percent. Then under three it goes to sixty - four point six percent. Professor B: Yeah, that's probably Aurora. PhD A: Yeah. Professor B: I mean {disfmarker} PhD A: So m I guess maybe they're error rates but they're, uh {disfmarker} they're really high. Professor B: I {disfmarker} I {disfmarker} I don't find that surpri PhD A: So {disfmarker} Professor B: I mean, we {disfmarker} W what's {disfmarker} what's some of the lower error rates on {disfmarker} on {disfmarker} on {disfmarker} uh, some of the higher error rates on, uh, {vocalsound} some of these w uh, uh, highly mismatched difficult conditions? What's a {disfmarker}? PhD D: Uh. Yeah, it's around fifteen to twenty percent. PhD A: Correct? PhD D: And the baseline, eh {disfmarker} PhD A: Accuracy? PhD D: Uh, error rate. Professor B: Yeah. PhD D: Twenty percent error rate, Professor B: Yeah. So twenty percent error rate on digits. PhD D: and {disfmarker} PhD A: Oh, oh, on digits. Professor B: So if you're doing {disfmarker} so if you're doing, PhD D: and {disfmarker} PhD A: Yeah. PhD D: On digits. PhD A: OK. Professor B: you know, PhD D: And this is so {disfmarker} so {disfmarker} still the baseline. Professor B: sixty - thousand {disfmarker} PhD D: Right? PhD A: Yeah. Professor B: Yeah, and if you're saying sixty - thousand word recognition, getting sixty percent error on some of these noise condition not at all surprising. PhD A: Yeah. PhD D: The baseline is sixty percent also on digits, PhD A: Oh, is it? PhD D: on the m more {pause} mismatched conditions. PhD A: OK. Professor B: Yeah. PhD D: So. PhD A: So, yeah, that's probably what it is then. Yeah. So they have a lot of different conditions that they're gonna be filling out. Professor B: It's a bad sign when you {disfmarker} looking at the numbers, you can't tell whether it's accuracy or error rate. PhD A: Yeah. Yeah. It's {disfmarker} it's gonna be hard. Um, they're {disfmarker} I I'm still waiting for them to {pause} release the, um, {vocalsound} multi - CPU version of their scripts, cuz right now their script only handles processing on a single CPU, which will take a really long time to run. So. But their s Professor B: This is for the training? PhD A: Uh {disfmarker} I beli Yes, for the training {pause} also. And, um, they're supposed to be coming out with it any time, Professor B: OK. PhD A: the multi - CPU one. So, as soon as they get that, then I'll {disfmarker} I'll grab those too Professor B: OK. PhD A: and so w Professor B: Yeah. Cuz we have to get started, PhD A: Yeah. Professor B: cuz it's {disfmarker} cuz, uh, PhD A: Yeah. I'll go ahead and try to run it though with just the single CPU one, Professor B: if the {disfmarker} PhD A: and {disfmarker} I {disfmarker} they {disfmarker} they, {vocalsound} um, released like a smaller data set that you can use that only takes like sixteen hours to train and stuff. So I can {disfmarker} I can run it on that just to make sure that the {disfmarker} {vocalsound} the thing works and everything. Professor B: Oh! Good. Yeah. Cuz we'll {disfmarker} Grad E: Hmm. Professor B: I guess the actual evaluation will be in six weeks or something. So. Is that about right {pause} you think? PhD D: Uh, we don't know yet, I {disfmarker} I think. Professor B: Really, we don't know? PhD D: Uh - huh. Um. PhD A: It wasn't on the conference call this morning? Professor B: Hmm. PhD D: No. PhD A: Hmm. Did they say anything on the conference call {pause} about, um, how the {pause} Wall Street Journal part of the test was going to be {pause} run? Because I {disfmarker} I thought I remembered hearing that some sites {vocalsound} were saying that they didn't have the compute to be able to run the Wall Street Journal stuff at their place, PhD D: No. Mmm. PhD A: so there was some talk about having Mississippi State run {pause} the systems for them. And I {disfmarker} Did {disfmarker} did that come up at all? PhD D: Uh, no. Well, this {disfmarker} first, this was not the point at all of this {disfmarker} the meeting today PhD A: Oh, OK. PhD D: and, Professor B: Some PhD D: uh, frankly, I don't know because I d {comment} didn't read also the {pause} most recent mails about {vocalsound} the large - vocabulary task. But, {vocalsound} uh, did you {disfmarker} do you still, uh, get the mails? You're not on the mailing list or what? PhD A: Hmm - mm. The only, um, mail I get is from Mississippi State {disfmarker} PhD D: Uh - huh. PhD A: so {disfmarker} PhD D: Oh, yeah. So we should have a look at this. PhD A: about their system. I {disfmarker} I don't get any {pause} mail about {disfmarker} Professor B: I have to say, there's uh something funny - sounding about saying that one of these big companies doesn't have enough cup compute power do that, so they're having to have it done by Mississippi State. PhD A: Yeah. Professor B: It just {disfmarker} {vocalsound} just sounds funny. PhD A: Yeah. It does. Professor B: But, PhD A: Yeah. I'm {disfmarker} I'm wondering about that Professor B: anyway. PhD A: because there's this whole issue about, you know, simple tuning parameters, like word insertion penalties. PhD D: Mm - hmm. PhD A: And {pause} whether or not those are going to be tuned or not, and {disfmarker} {comment} So. PhD D: Mm - hmm. PhD A: I mean, it makes a big difference. If you change your front - end, you know, the scale is completely {disfmarker} can be completely different, so. It seems reasonable that that at least should be tweaked to match the front - end. But {disfmarker} PhD D: You didn't get any answer from {pause} Joe? PhD A: I did, but Joe {pause} said, you know," what you're saying makes sense PhD D: Uh - huh. PhD A: and {pause} I don't know" . So he doesn't know what the answer is. PhD D: Uh - huh. PhD A: I mean, that's th We had this back and forth a little bit about, {vocalsound} you know, are sites gonna {disfmarker} are you gonna run this data for different sites? And, well, if {disfmarker} if Mississippi State runs it, then maybe they'll do a little optimization on that {pause} parameter, and, uh {disfmarker} But then he wasn't asked to run it for anybody. So i it's {disfmarker} it's just not clear yet what's gonna happen. PhD D: Mm - hmm. PhD A: Uh, he's been putting this stuff out on their web site and {disfmarker} for people to grab but I haven't heard too much about what's happening. Professor B: So it could be {disfmarker} I mean, Chuck and I had actually talked about this a couple times, and {disfmarker} and {disfmarker} over some lunches, I think, {vocalsound} that, um, {vocalsound} one thing that we might wanna do {disfmarker} The - there's this question about, you know, what do you wanna scale? Suppose y you can't adjust {vocalsound} these word insertion penalties and so forth, so you have to do everything at the level of the features. What could you do? And, uh, one thing I had suggested at an earlier time was maybe some sort of scaling, some sort of root or {disfmarker} or something of the, um, {vocalsound} uh, features. But the problem with that is that isn't quite the same, it occurred to me later, because what you really want to do is scale the, uh, @ @ {comment} the range of the likelihoods rather than {disfmarker} PhD D: Nnn, the dist Yeah. Professor B: But, {vocalsound} what might get at something similar, it just occurred to me, is kind of an intermediate thing {disfmarker} is because we do this strange thing that we do with the tandem system, at least in that system what you could do {vocalsound} is take the, um, {vocalsound} uh, values that come out of the net, which are something like log probabilities, and scale those. And then, uh, um {disfmarker} {pause} then at least those things would have the right values or the right {disfmarker} the right range. And then that goes into the rest of it and then that's used as observations. So it's {disfmarker} it's, {vocalsound} um, another way to do it. PhD D: Mm - hmm. Mm - hmm. But, these values are not directly used as probabilities anyway. Professor B: I know they're not. PhD D: So there are {disfmarker} there is {disfmarker} Professor B: I know they're not. But {disfmarker} but, you know {disfmarker} So because what we're doing is pretty strange and complicated, we don't really know what the effect is {pause} at the other end. PhD D: Uh - huh. Mm - hmm. Professor B: So, {vocalsound} um, {pause} my thought was maybe {disfmarker} I mean, they're not used as probabilities, but the log probabilities {disfmarker} we're taking advantage of the fact that something like log probabilities has more of a Gaussian shape than Gaus - than {vocalsound} probabilities, and so we can model them better. So, {pause} in a way we're taking advantage of the fact that they're probabilities, because they're this quantity that looks kind of Gaussian when you take it's log. So, {comment} {vocalsound} uh, maybe {disfmarker} maybe it would have a {disfmarker} a reasonable effect to do that. PhD D: Mm - hmm. Professor B: I d I don't know. But, {pause} I mean, I guess we still haven't had a {disfmarker} {vocalsound} a ruling back on this. And we may end up being in a situation where we just you know really can't change the {vocalsound} word insertion penalty. But the other thing we could do {vocalsound} is {disfmarker} also we could {disfmarker} I mean, this {disfmarker} this may not help us, {vocalsound} uh, in the evaluation but it might help us in our understanding at least. We might, {vocalsound} just run it with different insper insertion penalties, and show that, uh," well, OK, not changing it, {vocalsound} playing the rules the way you wanted, we did this. But in fact if we did that, it made a {disfmarker} {pause} a big difference." PhD A: I wonder if it {disfmarker} it might be possible to, uh, simulate the back - end with some other system. So we {disfmarker} we get our f front - end features, and then, uh, as part of the process of figuring out the scaling of these features, {comment} you know, if we're gonna take it to a root or to a power or something, {comment} {vocalsound} we have some back - end that we attach onto our features that sort of simulates what would be happening. Professor B: Mm - hmm. PhD A: Um, Professor B: And just adjust it until it's the best number? PhD A: and just adjust it until that {disfmarker} our l version of the back - end, uh, decides that {disfmarker} that {disfmarker} Professor B: Well, we can probably use the real thing, can't we? And then jus just, uh, {vocalsound} use it on a reduced test set or something. PhD A: Yeah. Oh, yeah. That's true. Professor B: Yeah. PhD A: And then we just use that to determine some scaling factor that we use. Professor B: Yeah. So I mean, I I think that that's a reasonable thing to do and the only question is what's the actual knob that we use? PhD A: Mm - hmm. Professor B: And the knob that we use should {disfmarker} uh, uh, unfortunately, like I say, I don't know the analytic solution to this cuz what we really want to do is change the scale of the likelihoods, PhD A: Mm - hmm. Professor B: not the cha not the scale of the {disfmarker} {vocalsound} the {pause} observations. But {disfmarker} but, uh {disfmarker} PhD D: Mm - hmm. PhD A: Yeah. Grad E: Out of curiosity, what {disfmarker} what kind of recognizer {pause} is the one from Mississippi State? PhD A: Uh, w what do you mean when you say" what kind" ? Grad E: Is it {disfmarker}? Um, is it like a {pause} Gaussian mixture model? PhD A: Yeah. Gaussian mixture model. Grad E: OK. PhD A: It's the same system that they use {pause} when they participate in the Hub - five evals. It's a, {vocalsound} um {disfmarker} sort of {pause} came out of, uh {disfmarker} uh, looking a lot like HTK. I mean, they started off with {disfmarker} um, when they were building their system they were always comparing to HTK to make sure they were getting similar results. And so, {vocalsound} it's a Gaussian mixture system, uh {disfmarker} Professor B: Do they have the same sort of mix - down sort of procedure, where they {vocalsound} start off with a small number of some things PhD A: I don't know. Yeah. And then {pause} divide the mixtures in half. Professor B: and {disfmarker}? Yeah. PhD A: I don't know if they do that. I'm not really sure. Professor B: Yeah. Grad E: Hmm. Professor B: D Do you know what kind of tying they use? Are they {disfmarker} they sort of {disfmarker} some sort of {disfmarker} a bunch of Gaussians that they share across everything? Or {disfmarker} {vocalsound} or if it's {disfmarker}? PhD A: Yeah, th I have {disfmarker} I {disfmarker} I {disfmarker} I don't have it up here but I have a {disfmarker} {pause} the whole system description, that describes exactly what their {pause} system is Professor B: OK. PhD A: and I {disfmarker} I'm not sure. But, um {disfmarker} Professor B: OK. PhD A: It's some kind of a mixture of Gaussians and, {vocalsound} uh, clustering and, uh {disfmarker} They're {disfmarker} they're trying to put in sort of all of the standard features that people use nowadays. Grad E: Mm - hmm. Professor B: So the other, uh, Aurora thing maybe is {disfmarker} I I dunno if any of this is gonna {vocalsound} {pause} come in in time to be relevant, but, uh, we had talked about, uh, {comment} Guenter {vocalsound} playing around, uh, uh, over in Germany PhD D: Mm - hmm. Professor B: and {disfmarker} and, @ @ {comment} uh, {pause} possibly coming up with something {vocalsound} that would, uh, {pause} uh, fit in later. Uh, I saw that other mail where he said that he {disfmarker} {vocalsound} uh, it wasn't going to work for him to do CVS. PhD D: Yeah. Yeah. So now he has a version of the software. Professor B: So he just has it all sitting there. Yeah. PhD D: Yeah. Um {disfmarker} Mm - hmm. Professor B: So if he'll {disfmarker} he might work on improving the noise estimate or on {vocalsound} some histogram things, or {disfmarker} PhD D: Yeah. Mm - hmm. Professor B: Yeah. I just saw the Eurospeech {disfmarker} We {disfmarker} we didn't talk about it at our meeting but I just saw the {disfmarker} just read the paper. Someone, I forget the name, {comment} and {disfmarker} and Ney, uh, about histogram equalization? Did you see that one? PhD D: Um, it was a poster. Or {disfmarker} Professor B: Yeah. I mean, I just read the paper. PhD D: Yeah. Professor B: I didn't see the poster. PhD D: Yeah. Um {disfmarker} {vocalsound} It was something {pause} similar to n {vocalsound} on - line normalization finally {disfmarker} I mean, in {vocalsound} the idea of {disfmarker} of normalizing {disfmarker} Professor B: Yeah. But it's a little more {disfmarker} it {disfmarker} it's a little finer, right? So they had like ten quantiles PhD D: Yeah. Professor B: and {disfmarker} {vocalsound} and they adjust the distribution. PhD D: Right. Professor B: So you {disfmarker} you have the distributions from the training set, PhD D: N Professor B: and then, uh {disfmarker} So this is just a {disfmarker} a histogram of {disfmarker} of {vocalsound} the amplitudes, I guess. Right? And then {disfmarker} {vocalsound} Um, people do this in image processing some. PhD D: Mm - hmm. Professor B: You have this kind of {disfmarker} {vocalsound} of histogram of {disfmarker} of levels of brightness or whatever. And {disfmarker} and {disfmarker} and then, {vocalsound} when you get a new {disfmarker} new thing that you {disfmarker} you want to adjust to be {pause} better in some way, {vocalsound} you adjust it so that the histogram of the new data looks like the old data. PhD A: Hmm. Professor B: You do this kind of {vocalsound} piece - wise linear or, {vocalsound} uh, some kind of piece - wise approximation. They did a {disfmarker} uh one version that was piece - wise linear and another that had a power law thing between them {disfmarker} {vocalsound} between the {pause} points. And, uh, they said they s they sort of see it in a way as s for the speech case {comment} {disfmarker} as being kind of a generalization of spectral subtraction in a way, because, you know, in spectral subtraction you're trying to {vocalsound} get rid of this excess energy. Uh, you know, it's not supposed to be there. Uh {disfmarker} {vocalsound} and, uh, this is sort of {pause} {vocalsound} adjusting it for {disfmarker} for a lot of different levels. And then they have s they have some kind of, {vocalsound} uh, {pause} a floor or something, Grad E: Hmm. Professor B: so if it gets too low you don't {disfmarker} don't do it. PhD A: Hmm. Professor B: And they {disfmarker} they claimed very nice results, PhD D: Mm - hmm. PhD A: So is this a histogram across different frequency bins? Professor B: and {disfmarker} PhD A: Or {disfmarker}? Professor B: Um, I think this i You know, I don't remember that. Do you remember {disfmarker}? PhD D: I think they have, yeah, different histograms. I uh {disfmarker} Something like one per {pause} frequency band, Professor B: One {disfmarker} PhD A: So, one histogram per frequency bin. Professor B: One per critical {disfmarker} PhD D: or {disfmarker} But I did {disfmarker} Yeah, I guess. PhD A: And that's {disfmarker} PhD D: But I should read the paper. I just went {pause} through the poster quickly, Professor B: Yeah. PhD A: So th Professor B: And I don't remember whether it was {pause} filter bank things PhD A: Oh. PhD D: and I didn't {disfmarker} Professor B: or whether it was FFT bins PhD A: Huh. Professor B: or {disfmarker} PhD A: And {disfmarker} and that {disfmarker} that, um, {pause} histogram represents {pause} the {pause} different energy levels that have been seen at that {pause} frequency? Professor B: I don't remember that. And how often they {disfmarker} you've seen them. Yeah. PhD A: Uh - huh. Grad E: Hmm. Professor B: Yeah. And they do {disfmarker} they said that they could do it for the test {disfmarker} So you don't have to change the training. You just do a measurement over the training. And then, uh, for testing, uh, you can do it for one per utterance. Even relatively short utterances. And they claim it {disfmarker} it works pretty well. PhD A: So they, uh {disfmarker} Is the idea that you {disfmarker} you run a test utterance through some histogram generation thing and then you compare the histograms and that tells you {vocalsound} what to do to the utterance to make it more like {disfmarker}? Professor B: I guess in pri Yeah. In principle. PhD A: I see. Professor B: I didn't read carefully how they actually implemented it, PhD A: Hmm. Yeah. Professor B: whether it was some, {vocalsound} uh, on - line thing, or whether it was a second pass, or what. But {disfmarker} but they {disfmarker} {vocalsound} That {disfmarker} that was sort of the idea. PhD A: Hmm. Professor B: So that {disfmarker} that seemed, you know, different. We're sort of curious about, uh, what are some things that are, u u um, {vocalsound} @ @ {comment} {pause} conceptually quite different from what we've done. PhD A: Mm - hmm. Professor B: Cuz we {disfmarker} you know, one thing that w that, uh, Stephane and Sunil seemed to find, {vocalsound} uh, was, you know, they could actually make a unified piece of software that handled a range of different things that people were talking about, and it was really just sort of setting of different {pause} constants. And it would turn, you know, one thing into another. It'd turn Wiener filtering into spectral subtraction, or whatever. But there's other things that we're not doing. So, we're not making any use of pitch, uh, uh, which again, might {disfmarker} might be important, uh, because the stuff between the harmonics is probably a schmutz. And {disfmarker} and the, {vocalsound} uh, transcribers will have fun with that. Uh {disfmarker} {vocalsound} And, um, the, uh, stuff at the harmonics isn't so much. And {disfmarker} and, uh {disfmarker} And we there's this overall idea of really sort of matching the {disfmarker} the hi distributions somehow. Uh, not just, um, {vocalsound} um {disfmarker} not just subtracting off your estimate of the noise. So. So I guess, uh, {vocalsound} Guenter's gonna play around with some of these things now over this next {pause} period, PhD D: Uh, I dunno. Professor B: or {disfmarker}? PhD D: I don't have feedback from him, but Professor B: Yeah. PhD D: I guess he's gonna, maybe {disfmarker} Professor B: Well, he's got it anyway, so he can. PhD D: Yeah. Professor B: So potentially if he came up with something that was useful, like a diff a better noise estimation module or something, he could ship it to you guys u up there PhD D: Uh - huh. Professor B: and PhD D: Yeah. Professor B: we could put it in. PhD D: Mm - hmm. {vocalsound} Mm - hmm. Professor B: Yeah. Yeah. So, that's good. So, why don't we just, uh, um {disfmarker} I think starting {disfmarker} {pause} starting a w couple weeks from now, especially if you're not gonna be around for a while, we'll {disfmarker} we'll be shifting more over to some other {disfmarker} {vocalsound} other territory. But, uh, uh, {comment} uh, n not {disfmarker} not so much in this meeting about Aurora, but {disfmarker} but, uh, uh, maybe just, uh, quickly today about {disfmarker} maybe you could just say a little bit about what you've been talking about with Michael. And {disfmarker} and then Barry can say something about {pause} what {comment} {disfmarker} what we're talking about. Grad C: OK. So Michael Kleinschmidt, who's a PHD student from Germany, {vocalsound} showed up this week. He'll be here for about six months. And he's done some work using {vocalsound} an auditory model {pause} of, um, {vocalsound} human hearing, and {pause} using that f uh, to generate speech recognition features. And {pause} he did {vocalsound} work back in Germany {vocalsound} with, um, a toy recognition system {vocalsound} using, um, isolated {vocalsound} digit recognition {vocalsound} as the task. It was actually just a single - layer neural network {vocalsound} that classified words {disfmarker} classified digits, {vocalsound} in fact. Um, and {pause} he tried that on {disfmarker} I think on some Aurora data and got results that he thought {pause} seemed respectable. And he w he's coming here to u u use it on a {vocalsound} uh, a real speech recognition system. So I'll be working with him on that. And, um, maybe I should say a little more about these features, although I don't understand them that well. The {disfmarker} I think it's a two - stage idea. And, um, {vocalsound} the first stage of these features correspond to what's called the peripheral {vocalsound} auditory system. And {vocalsound} I guess that is like {vocalsound} a filter bank with a compressive nonlinearity. And {vocalsound} I'm - I'm not sure what we have @ @ in there that isn't already modeled in something like, {vocalsound} um, {pause} PLP. I should learn more about that. And then {vocalsound} the second stage {pause} is, um, {vocalsound} the most different thing, I think, from what we usually do. It's, um {disfmarker} {vocalsound} {vocalsound} it computes features which are, {vocalsound} um, {vocalsound} based on {disfmarker} sort of like based on diffe different w um, wavelet basis functions {vocalsound} used to analyze {vocalsound} the input. So th he uses analysis functions called {vocalsound} Gabor functions, um, {vocalsound} which have a certain {vocalsound} extent, um, {vocalsound} in time and in frequency. And {vocalsound} the idea is these are used to sample, {vocalsound} um, the signal in a represented as a time - frequency representation. So you're {pause} sampling some piece of this time - frequency plane. And, um, {vocalsound} that, {vocalsound} um, is {disfmarker} is interesting, cuz, {vocalsound} @ @ for {disfmarker} for one thing, you could use it, {vocalsound} um, in a {disfmarker} a multi - scale way. You could have these {disfmarker} instead of having everything {disfmarker} like we use a twenty - five millisecond or so analysis window, {vocalsound} typically, um, and that's our time scale for features, but you could {disfmarker} {vocalsound} using this, um, basis function idea, you could have some basis functions which have a lot longer time scale and, um, some which have a lot shorter, and {vocalsound} so it would be like {pause} a set of multi - scale features. So he's interested in, um {disfmarker} Th - this is {disfmarker} because it's, um {disfmarker} there are these different parameters for the shape of these {vocalsound} basis functions, {vocalsound} um {disfmarker} {vocalsound} there are a lot of different possible basis functions. And so he {disfmarker} {vocalsound} he actually does {vocalsound} an optimization procedure to choose an {disfmarker} {vocalsound} an optimal set of basis functions out of all the possible ones. PhD A: Hmm. H What does he do to choose those? Grad C: The method he uses is kind of funny {disfmarker} is, {comment} {vocalsound} um, {vocalsound} he starts with {disfmarker} he has a set of M of them. Um, he {disfmarker} and then {pause} he uses that to classify {disfmarker} I mean, he t he tries, um, {vocalsound} using {pause} just M minus one of them. So there are M possible subsets of this {vocalsound} length - M vector. He tries classifying, using each of the M {vocalsound} possible sub - vectors. PhD D: Hmm. Grad C: Whichever sub - vector, {vocalsound} um, works the {disfmarker} the best, I guess, he says {disfmarker} {vocalsound} the {disfmarker} the fe feature that didn't use was the most useless feature, Professor B: Y yeah. Gets thrown out. Yeah. Grad C: so we'll throw it out and we're gonna randomly select another feature {pause} from the set of possible basis functions. PhD A: Hmm! Professor B: Yeah. PhD A: So it's a {disfmarker} Professor B: So i so it's actuall PhD A: it's a little bit like a genetic algorithm or something in a way. Professor B: Well, it's {disfmarker} it's much simpler. Grad E: It's like a greedy {disfmarker} Professor B: But it's {disfmarker} but it's {disfmarker} uh, it's {disfmarker} there's a lot {disfmarker} number of things I like about it, let me just say. PhD A: Greedy. Professor B: So, first thing, well, you're absolutely right. I mean, {vocalsound} i i {nonvocalsound} in truth, {pause} both pieces of this are {disfmarker} have their analogies in stuff we already do. But it's a different take {vocalsound} at how to approach it and potentially one that's m maybe a bit more systematic than what we've done, uh, and a b a bit more inspiration from {disfmarker} from auditory things. So it's {disfmarker} so I think it's a neat thing to try. The primary features, {vocalsound} um, are in fact {disfmarker} Yeah, essentially, it's {disfmarker} it's, uh, you know, PLP or {disfmarker} or mel cepstrum, or something like that. You've {disfmarker} you've got some, {vocalsound} uh, compression. We always have some compression. We always have some {disfmarker} you know, the {disfmarker} the {disfmarker} the kind of filter bank with a kind of {vocalsound} {vocalsound} quasi - log scaling. Um, {vocalsound} if you put in {disfmarker} if you also include the RASTA in it {disfmarker} i RASTA {disfmarker} the filtering being done in the log domain {vocalsound} has an AGC - like, uh, characteristic, which, you know, people typi typically put in these kind of, {vocalsound} uh, {pause} um, {vocalsound} uh, auditory front - ends. So it's very, very similar, uh, but it's not exactly the same. Um, I would agree that the second one is {disfmarker} is somewhat more different but, {vocalsound} um, it's mainly different in that the things that we have been doing like that have been {disfmarker} {vocalsound} um, had a different kind of motivation and have ended up with different kinds of constraints. So, for instance, if you look at the LDA RASTA stuff, {vocalsound} you know, basically what they do is they {disfmarker} they look at the different eigenvectors out of the LDA and they form filters out of it. Right? And those {pause} filters have different, uh, kinds of temporal extents and temporal characteristics. And so in fact they're multi - scale. But, they're not sort of systematically multi - scale, like" let's start here and go to there, and go to there, and go to there" , and so forth. It's more like, {vocalsound} you run it on this, you do discriminant analysis, and you find out what's helpful. Grad C: I it's multi - scale because you use several of these in parallel, Professor B: Yeah. They use several of them. Grad C: is that right? Of {disfmarker} Professor B: Yeah. Grad C: OK. Professor B: Uh, I mean, you don't have to but {disfmarker} but {disfmarker} but, uh, Hynek has. Um, but it's also, uh {disfmarker} Hyn - when Hynek's had people do this kind of LDA analysis, they've done it on frequency direction and they've done it on the time direction. I think he may have had people sometimes doing it on both simultaneously {disfmarker} some two - D {disfmarker} and that would be the closest to these Gabor function kind of things. Uh, but I don't think they've done that much of that. And, uh, the other thing that's interesting {disfmarker} the {disfmarker} the, uh {disfmarker} the feature selection thing, it's a simple method, but I kinda like it. Um, {vocalsound} there's a {disfmarker} {pause} a old, old method for feature selection. I mean, {pause} eh, uh, I remember people referring to it as old when I was playing with it twenty years ago, so I know it's pretty old, uh, called Stepwise Linear Discriminant Analysis in which you {disfmarker} which {disfmarker} I think it's used in social sciences a lot. So, you {disfmarker} you {disfmarker} you {disfmarker} you pick the best feature. And then {vocalsound} you take {disfmarker} y you find the next feature that's the best in combination with it. And then so on and so on. And what {disfmarker} what Michael's describing seems to me much, much better, because the problem with the stepwise discriminant analysis is that you don't know that {disfmarker} you know, if you've {vocalsound} picked the right set of features. Just because something's a good feature doesn't mean that you should be adding it. So, {vocalsound} um, {pause} uh, here at least you're starting off with all of them, and you're {vocalsound} throwing out useless features. I think that's {disfmarker} that seems, uh {disfmarker} {vocalsound} that seems like a lot better idea. Uh, you're always looking at things in combination with other features. Um, so the only thing is, of course, there's this {disfmarker} this artificial question of {disfmarker} of, uh, {vocalsound} exactly how you {disfmarker} how you a how you assess it and if {disfmarker} if your order had been different in throwing them out. I mean, it still isn't necessarily really optimal, but it seems like a pretty good heuristic. So I th I think it's {disfmarker} it's {disfmarker} I think it's kinda neat stuff. Grad E: Hmm. Professor B: And {disfmarker} and {disfmarker} and, uh, the thing that I wanted to {disfmarker} to add to it also was to have us use this in a multi - stream way. Grad E: Hmm. Professor B: Um, so {disfmarker} so that, um, {vocalsound} when you come up with these different things, {vocalsound} and these different functions, {vocalsound} you don't necessarily just put them all into one huge vector, but perhaps {vocalsound} you {vocalsound} have some of them in one stream and some of them in another stream, and so forth. And, um, um, {comment} um {disfmarker} And we've also talked a little bit about, uh, {vocalsound} uh, Shihab Shamma's stuff, in which {vocalsound} you {disfmarker} the way you look at it is that there's these different mappings and some of them emphasize, uh, upward moving, {vocalsound} uh, energy and fre and frequency. And some are emphasizing downward and {vocalsound} fast things and slow things and {disfmarker} and {pause} so forth. So. So there's a bunch of stuff to look at. But, uh, I think we're sorta gonna start off with what {vocalsound} he, uh, came here with and branch out {disfmarker} {vocalsound} branch out from there. And his advisor is here, too, {vocalsound} at the same time. So, he'll be another {pause} interesting source of {pause} wisdom. Grad E: Hmm. Professor B: So. Grad E: As {disfmarker} as we were talking about this I was thinking, {vocalsound} um, {vocalsound} whether there's a relationship between {disfmarker} {vocalsound} um, {vocalsound} {vocalsound} between Michael's approach to, uh, some {disfmarker} some sort of optimal brain damage or optimal brain surgeon on the neural nets. Professor B: Yeah. Grad C: Hmm. Grad E: So, like, if we have, um {disfmarker} we have our {disfmarker} we have our RASTA features and {disfmarker} and presumably the neural nets are {disfmarker} are learning some sort of a nonlinear mapping, {vocalsound} uh, from the {disfmarker} the {disfmarker} the features {vocalsound} to {disfmarker} to this {disfmarker} this probability posterior space. Professor B: Mm - hmm. Grad E: Right? And, um {disfmarker} {vocalsound} {vocalsound} and each of the hidden units is learning some sort of {disfmarker} some sort of {disfmarker} some sort of pattern. Right? And it could be, like {disfmarker} {vocalsound} like these, um {disfmarker} these auditory patterns that Michael {pause} is looking at. And then when you're looking at the {disfmarker} {vocalsound} the, uh, {pause} um, {vocalsound} the best features, {vocalsound} you know, you can take out {disfmarker} you can do the {disfmarker} do this, uh, brain surgery by taking out, {vocalsound} um, hidden units that don't really help at all. Professor B: Mm - hmm. Or the {disfmarker} or features. Grad E: And this is k sorta like {disfmarker} Professor B: Right? Grad E: Yeah. Professor B: I mean, y actually, you make me think a {disfmarker} a very important point here is that, um, {vocalsound} if we a again try to look at how is this different from what we're already doing, {vocalsound} uh, there's a {disfmarker} a, uh {disfmarker} {vocalsound} a nasty argument that could be made th that it's {disfmarker} it's not different at {disfmarker} at all, because, uh {disfmarker} if you ignore the {disfmarker} the selection part because we are going into a {disfmarker} a very powerful, {vocalsound} uh, nonlinearity that, uh, in fact is combining over time and frequency, and is coming up with its own {disfmarker} you know, better than Gabor functions its, you know, neural net functions, Grad E: Mm - hmm. Professor B: its {disfmarker} {comment} {vocalsound} whatever it finds to be best. Grad C: Professor B: Um, so you could argue that in fact it {disfmarker} But I {disfmarker} I don't actually believe that argument because I know that, um, {vocalsound} you can, uh {disfmarker} computing features is useful, even though {pause} in principle you haven't {pause} {vocalsound} added anything {disfmarker} in fact, you subtracted something, from the original waveform {disfmarker} You know, uh, if you've {disfmarker} you've processed it in some way you've typically lost something {disfmarker} some information. And so, {vocalsound} you've lost information and yet it does better with {disfmarker} {vocalsound} with features than it does with the waveform. So, uh, I {disfmarker} I know that i sometimes it's useful to {disfmarker} {pause} to constrain things. So that's {vocalsound} why it really seems like the constraint {disfmarker} in {disfmarker} in all this stuff it's the constraints that are actually what matters. Because if it wasn't {pause} the constraints that mattered, then we would've completely solved this problem long ago, because long ago we already knew how to put waveforms into powerful statistical mechanisms. So. PhD D: Yeah. Well, if we had infinite processing power and {pause} data, {comment} I guess, using the waveform could {disfmarker} Grad E: Right. Professor B: Yeah Uh, then it would work. Yeah, I agree. Yeah. There's the problem. PhD D: So, that's {disfmarker} Professor B: Yeah. Then it would work. But {disfmarker} but, I mean, i it's {disfmarker} {vocalsound} With finite {pause} of those things {disfmarker} I mean, uh, we {disfmarker} we have done experiments where we literally have put waveforms in and {disfmarker} and {disfmarker} and, uh, PhD D: Mm - hmm. Professor B: we kept the number of parameters the same and so forth, and it used a lot of training data. And it {disfmarker} and it {disfmarker} it, uh {disfmarker} not infinite but a lot, and then compared to the number parameters {disfmarker} and it {disfmarker} it, uh {disfmarker} it just doesn't do nearly as well. So, anyway the point is that you want to suppress {disfmarker} PhD D: Mm - hmm. Professor B: it's not just having the maximum information, you want to suppress, {vocalsound} uh, the aspects of the input signal that are not helpful for {disfmarker} for the discrimination you're trying to make. So. So maybe just briefly, uh {disfmarker} Grad E: Well, that sort of segues into {pause} what {disfmarker} what I'm doing. Professor B: Yeah. Grad E: Um, {vocalsound} so, uh, the big picture is k um, {vocalsound} come up with a set of, {vocalsound} uh, intermediate categories, then build intermediate category classifiers, then do recognition, and, um, improve speech recognition in that way. Um, so right now I'm in {disfmarker} in the phase where {vocalsound} I'm looking at {disfmarker} at, um, deciding on a initial set of intermediate categories. And {vocalsound} I'm looking {vocalsound} for data data - driven {pause} methods that can help me find, {vocalsound} um, a set of intermediate categories {vocalsound} of speech that, uh, will help me to discriminate {pause} later down the line. And one of the ideas, {vocalsound} um, that was to take a {disfmarker} take a neural net {disfmarker} train {disfmarker} train an ordinary neural net {vocalsound} to {disfmarker} {vocalsound} uh, to learn the posterior probabilities of phones. And so, um, at the end of the day you have this neural net and it has hidden {disfmarker} {vocalsound} hidden units. And each of these hidden units is {disfmarker} {vocalsound} um, is learning some sort of pattern. And so, um, what {disfmarker} what are these patterns? PhD A: Hmm. Grad E: I don't know. Um, and I'm gonna to try to {disfmarker} {vocalsound} to look at those patterns {vocalsound} to {disfmarker} to see, {vocalsound} um, {vocalsound} from those patterns {disfmarker} uh, presumably those are important patterns for discriminating between phone classes. And maybe {disfmarker} {vocalsound} maybe some, uh, intermediate categories can come from {vocalsound} just looking at the patterns of {disfmarker} {vocalsound} um, that the neural net learns. Professor B: Be - before you get on the next part l let me just point out that s there's {disfmarker} there's a {disfmarker} a pretty nice {comment} {vocalsound} relationship between what you're talking about doing and what you're talking about doing there. Right? Grad E: Yeah. Professor B: So, {vocalsound} it seems to me that, you know, if you take away the {disfmarker} the {disfmarker} {pause} the difference of this {pause} primary features, {vocalsound} and, say, you use {disfmarker} as we had talked about maybe doing {disfmarker} you use P - RASTA - PLP or something for the {disfmarker} the primary features, {vocalsound} um, then this feature discovery, {pause} uh, uh, thing {vocalsound} is just what he's talking about doing, too, except that he's talking about doing them in order to discover {pause} intermediate categories that correspond {vocalsound} to these {disfmarker} uh, uh, what these sub - features are {disfmarker} are {disfmarker} are {disfmarker} are showing you. And, um, {vocalsound} the other difference is that, um, {vocalsound} he's doing this in a {disfmarker} in a multi - band setting, which means that he's constraining himself {vocalsound} to look across time in some f relatively limited, uh, uh, spectral extent. Right? And whereas in {disfmarker} in this case you're saying" let's just do it unconstrained" . So they're {disfmarker} they're really pretty related and maybe they'll be {disfmarker} at some point where we'll see the {disfmarker} the connections a little better and {vocalsound} connect them. Grad C: Hmm. Grad E: Mm - hmm. Um. Yeah, so {disfmarker} so that's the {disfmarker} that's the first part {disfmarker} uh, one {disfmarker} one of the ideas to get at some {disfmarker} {vocalsound} some patterns of intermediate categories. Um, {vocalsound} the other one {pause} was, {vocalsound} um, to, {vocalsound} uh, come up with a {disfmarker} a {disfmarker} a model {disfmarker} {comment} um, a graphical model, {vocalsound} that treats {pause} the intermediate categories {vocalsound} as hidden {disfmarker} hidden variables, latent variables, that we don't know anything about, but that through, {vocalsound} um, s statistical training and the EM algorithm, {vocalsound} um, at the end of the day, {vocalsound} we have, um {disfmarker} we have learned something about these {disfmarker} these latent, um {disfmarker} latent variables which happen to correspond to {vocalsound} intermediate categories. Um. {vocalsound} {nonvocalsound} Yeah, and so those are the {disfmarker} the two directions that I'm {disfmarker} I'm looking into right now. And, uh, {vocalsound} um {disfmarker} {vocalsound} {vocalsound} Yeah. I guess that's {disfmarker} that's it. Professor B: OK. Should we do our digits and get ou get our treats? Grad E: Oh, tea time? Professor B: Yeah. It's kind of like, you know, the little rats with the little thing dropping down to them. PhD A: That's ri Professor B: We do the digits and then we get our treats. Grad E: Oops. PhD A: OK.
PhD D expressed that the resynthesized version of the speech that LPC gave was not the same as speech. Though, the excitations were more clear in the resynthesized version.
What did PhD D think about LPC?