Interview with creator of The Virtual Flute

Andrew Botros was Engineers Australia’s Young Engineer of the Year in 2006 for his contributions to cochlear implants, music acoustics, machine intelligence and the engineering community, and has spoken internationally in these fields. Andrew is also a University Medallist, a winner of Australia’s Siemens Prize for Innovation and the Australian Acoustical Society’s Excellence in Acoustics Award. Andrew is a recipient of the Future Summit’s Australian Leadership Award, a judging panellist for the Australian Good Design Awards, and a graduate of public policy and administration at the ANU, where he conducted research in digital government and open innovation.

Hi Andrew, thanks for your time today. Would you give us a brief intro about yourself?

So I'm Andrew Botros, former engineer and a former student of the University of New South Wales. I did my undergraduate thesis in computer engineering, although at the School of Physics, which was unusual. I did a music project to build a computer simulation of the flute for the benefit of musicians, not just for intellectual stimulation. That's how I met you, Jane, as the flute player who could give advice and also try things out - so I would have met you in the year 2000.

What exactly is The Virtual Flute?

The Virtual Flute is, easily, the biggest fingering book for the flute known to man. Has never been attempted again in the fourteen years that it's been public. And it's comprehensive in the sense that a computer simulation will go through every possible fingering, of which there are tens of thousands, and make a fairly good prediction of what notes it will play, what multiphonics it can play and also the quality of the notes - being its intonation and timbre.

Who is The Virtual Flute useful for?

As time has gone on, it's tended to be of most use to those of the more advanced end of their flute playing - especially those who are particularly good at and have an interest in contemporary music. So The Virtual Flute's been credited in flute compositions, which has been, a great credit to the project itself. So, for example, Richard Barrett, in the past few years has had two compositions where at the footnote of the cover page is, "Thanks to The Virtual Flute", which has been a great compliment. And they have generally been commissioned for, or by, very, very skilled flutists who are interested in contemporary repertoire.

Is Richard Barrett the only composer you know of that has credited The Virtual Flute?

His compositions are the only ones that I know directly because they came up in web searches - I wasn't even informed of them. In fact, if we're speaking freely, one of the great regrets of my recent life is that one of those great compositions was premiered last year in London and I was in London at the time and had no idea and only found out a month later that I could've gone to the world premier of a piece of work dedicated to, in some part, The Virtual Flute.

Do you know of any flute players who use The Virtual Flute?

Every so often I get questions from flutists, composers, and researchers from around the world. Some musicians are quite memorable - I once corresponded with the flutist from a New York "collective of composers and performers" called Anti-Social Music. I bumped into one of the flautists of the Sydney Symphony Orchestra, and she was not aware of the work but that's OK. Kathleen Gallagher, a Sydney contemporary flautist often uses The Virtual Flute. Robert Dick is certainly aware of it - he wrote the most well-known fingering book in print. So he has been aware of The Virtual Flute since its very start. Compared to printed fingering books, The Virtual Flute is many, many times larger - and interactive.

How did you actually create The Virtual Flute?

So The Virtual Flute is built from physical measurements of the instrument. So at the physics lab at the University of New South Wales, very precise measurements of the acoustics of the flute are done. So there would be a machine that would be attached to the embouchure hole of the flute. It would put sound in and then listen to what sound would come out. And these sounds are not all that audible, but they're detected by a machine. But from that, you can tell what is the natural playing frequencies of the instrument from analysing the movement of air at the embouchure hole. So you do this for a number of different fingerings, and then you build up a data set where if you're going to have a go at trying to simulate it, you'd have a place to start. And so the first part of the project (and I spent probably almost the first twelve months doing this) was trying to get from physical principles, a fairly simple model, and trying to reproduce these measurements done in the lab. Once you've done that, you can get the computer to do every single fingering which no one's going to do in a manual way, because there are tens of thousands of those. So that's the first part of it. Then, now that you've got predictions of what the acoustics are like, I wanted to know if a musician would also have the same experience? And so that's where I met you and you stepped in and you would go through a number of fingerings and say which notes and which multiphonics you could play, and I would get the computer to learn using, literally, "learning algorithms", which actually are very widely talked about now in scientific circles and in the public. Machine learning is talked about all the time now. Fifteen years ago it was just a computer interest.

Is machine learning another way of saying artificial intelligence?

Machine learning is a branch of artificial intelligence. These days, it's almost the dominant branch. And so once you've got the physical measurements of the flute, and you know what the musician does with their playing, the computer learns how to put one to the next. And then once you've got all that information, you know what can be played, to a certain degree of accuracy, for every fingering. Put them on the web, it's searchable, and that's been The Virtual Flute for a very long time. And it's almost scary to think when The Virtual Flute was launched, Google was only three years old. It's a really interesting thought. But, by chance, the sort of software behind it that would just present the information is the same software that's still used today on the web and would be the same technology behind WordPress and a long list of other systems. So it's interesting how the web has moved and not moved in that time.

What sort of media coverage has The Virtual Flute received?

So the media coverage didn't come from the flute project itself. It came from the awards that were entered as part of that. So when I graduated, there was an annual prize for final year engineering projects - the big one at the time was the Siemens Prize for Innovation. They don't do it anymore but back then it was a very rich prize, it was a $25,000 prize. It was an Australia wide competition. You got a bit of money if you won the state entry, but that national prize was where it's at. And you were there that day because you were doing the demos as part of that. Once that was won, there was coverage from The Australian, the Daily Telegraph, a number of technical circles as well. The university got interested as well and put their own articles in the local paper, on their own media, etc. So that was my introduction to what the media's like. All that media experience made me think "how do you take a fairly complex project, though an interesting story, and do your very best to get it in the heads of someone who is from the media, and has little attention or time to dedicate to the story"? I'd get practice again, and again, and again on this. And so that was just a brilliant way to hone communication skills in a technical world. When I was doing the project, even in university circles, because it was a collaboration of music, computer science and physics, there was always most of the room who had no idea what I'm talking about. So I was very attuned to this, always, and I'd make no assumptions of what I was going to say and always explain everything from scratch. And that is a wonderful principle to hold on to for the rest of your life in whatever circles you interact with with others. So it was great practice - how do you take unfamiliar things and give it to other people? Even when you're in large organisations, you always think that if you're going to tell how your project is going to the bosses, it's easy to assume that they might just understand it already. Usually they don't. And so was a really good setting for just stepping back and thinking, "my audience needs a bit of background here and I'm going to give it to them - and in a short space of time because they're going to move on to something else very quickly." 

What sorts of things could a flute player do on The Virtual Flute website?

Moving from top to bottom on the website, you can click on the keys of the flute that are pressed for a note of your choice and it will tell you everything that that fingering will do - the notes it will play, its multiphonics, the exact intonation, at least predicted to a certain degree of accuracy, and the timbre of those notes. But that's just the start. More interesting still is to tell The Virtual Flute which notes you want to play and it will tell you which fingerings will play that note. More than that, if you want to explicitly have certain keys included, or not included, in that fingering, you can do that as well. So if you've got two notes in a passage of music that are side-by-side, that are difficult to play quickly together, like in a trill or just a quick passage, where the physical properties may not make it very good to have those two fingerings together, or you often split the notes or you get squeaks in between them, you can basically request a fingering that closely matches one of the fingerings and then see what options are available. And then finally you can say what multiphonics you want and it will see if there are predicted fingerings for that multiphonic - and the composers are really onto this all the time.

Would you consider creating The Virtual Flute, but for other instruments?

Since the flute project was done, it was always obvious to move on to other wind instruments and it never quite got off the ground. The funding, the interest, whatever, as a university project, it would never attract the right people to work on it and I'd since moved on to other things. But I did have a crack at the clarinet once and the accuracy wasn't good enough in the simulation to push on with it at that time. However, it just needed some extra work to see what needed to be done. The clarinet is different to the flute in that it is, obviously a reed instrument, for starters, and secondly, has very small holes that are quite difficult to model, as opposed to the flute holes which are quite large. And so if that could be worked on at some point in time, that would unlock those instrument and the same concepts would then apply. Secondly, it'd be nice to re-design the interface so that it's more accessible to the players that are not as advanced as those who found the most benefit. And also, as I mentioned, the web has moved on so much since it was created. The level of interaction through social media, etc, plugged into that website could be completely overhauled. Whereas now it has a very passive interface where you just get information out of it. So much has happened in the web since then - to be able to interact now with other users - it's just waiting for that to happen. So these are ideas, but who knows what the next couple years might bring? It has a new address now as well: flute.fingerings.info

Time has really passed since The Virtual Flute was launched hasn't it?

Yes, I started the project fifteen years ago, and it was launched fourteen years ago. Just passed the 1990s! So for a work for its day, it was quite a demonstration of capability for the web.

Do you have any good stories associated with the project?

I think the way the project got started is an interesting story. So engineering students get a list of projects that academics have put out for interesting projects to do for their final year. It's usually tacked onto their own research interests - they basically say, "Here are five things we want to work on." The thing that marked me out was I actually wanted to get out of the School of Computer Science. I'd been in there for four years, had had my challenges with the way things were run (academics, etc). Some experiences were great, some weren't so great, so I thought, "Wouldn't it be great to do a project that is not in the school?" I had an interest in music, but it was a very dormant interest in the sense that I used to go to concerts a lot, but not be able to play anything well. I played the piano a little bit, terribly - I dropped it when I was much younger. So when I saw on the list that there was a project in the School of Physics on music, I thought, "This might be interesting to have a go." But it sort of started my whole career in being comfortable in stepping out of the usual culture of what you do - absorbing another field to the point where you can make a contribution. I got the practice at doing this from that project. And also, because it wasn't an incremental change to something, it was a completely new project, I started with a blank page. So the rough idea had been set by Joe Wolfe and John Smith at the School of Physics (UNSW), but there was no anticipation of what it would look like, no anticipation of how successful it would be, and certainly no idea of what computing technology might be applied. And so that was a lot of responsibility to put on myself, but in a safe environment, because it was just a university project. And as the project went on, I'd have to solve problems of my own and I'd have to find answers to it without any direction and sometimes those solutions came by chance. So I came across machine learning sitting through a lecture midway through my final year in 2001, on a completely different topic, but the lecturer thought they'd just take a tangent and talk about it for a moment as something that they were doing at the time. And at that time, I was trying to figure out what to do with your data, Jane, of the responses to all these fingerings and how do I move from the physics to the actual music? It was a perfect possible way of going. Suddenly I'm now in the machine learning game to try and solve a problem on a project starting from a blank page. And once you get this sort of practice, you become quite confident in doing very different things and absorbing different concepts. And so, after I finished The Virtual Flute, I went to Cochlear for a number of years. I had that same confidence about trying different things and absorbing the medical field, no worries, making comments on how audiological procedures are done, no worries. And all that confidence came out of that Virtual Flute experience. Another interesting story is that it might have all come to nothing. Because I had a computing background, not a physics one (I mean, I studied a bit of physics, but it wasn't my field), I had made a mistake in one of the simulation programs because I had misread one of the graphs in the physics book. I hadn't understood quite what the units were and it wasn't absolutely clear in the book. So the simulation for the first twelve months was inaccurate but inaccurate by, not so much that you got ridiculous results, just a bit off. But, not looking great. It looks like you're in the right ball park, so it's not clear that you've made a mistake, it may just be that the model is just not good enough. Until one day I was sitting with Joe (Wolfe) late one afternoon, I'll never forget it, and he was just passing a look through and he found this wrong interpretation of units on this one graph and once I fixed that, it changed everything. So I had this whole lever arch binder full of results that I'd accumulated and tried different things with for twelve months, and it all went in the bin that afternoon, every single result. And so it wasn't the only thing I was doing in that twelve months, I was doing other things in the project - the web interface, the machine learning bit, etc. If it wasn't for that discovery, the project may never ever have got of the ground to the extent that it did.

How out were your results in the end?

They were in the ballpark to think that this may be a normal result but fingerings that are a lot of open and closed holes in combination are the most difficult to model. When they're all open, it's not so hard, when they're all closed it's not so hard. But when there's a mix of open and closed, suddenly you start seeing the discrepancies and they were quite large for some of the things we were trying to do. If just that one line of code... if that didn't get fixed, it would've just been an inaccurate project. So it was a great little real-world project; where things go wrong, where unknowns are there everywhere, there's ups and downs... I mean I was really depressed that day but I shouldn't have been - I should have been very, very happy that we moved on.

How has this led to other things that you're currently involved in?

I now own a business called Expressive Engineering and I've been doing it for four years. During my time at Cochlear where I been from 2002 to 2010, and I did my PhD there, half of my time there was spent arguing for new ideas. So in the music acoustics world, everyone is really friendly; everyone loves everyone else, they're all very collaborative because everyone's in it for the love of it. There's not a lot of money going around in music acoustics, at all. People are generally doing it as an interesting hobby on the core skill of whatever it might be. As you move into a commercial industry, whatever that may be, but especially a medical industry, everyone's got an opinion on direction and everyone will fight for it. This is very normal in organisations and so half my time at Cochlear was spent in technical disputes and the other half spent in pushing for ideas. Just having had that practice over the years, and also that experience of trying to explain The Virtual Flute to diverse audiences, handling the media, and then trying to persuade people to push it in certain directions, I thought by the end of that, when my PhD was done, I was probably more use to the world in that side of things, in the organisational communication of engineering, rather than just doing the technical work. So I don't necessarily focus on how to give a presentation - there's a wider mix. How should organizations communicate internally? One-on-one, to audiences, for maximum impact, maximum innovation, to take advantage of opportunities, to find them, how to structure arguments to support a project, the importance of messaging... All those sort of things lead to the effectiveness of projects and technical people often neglect them because it's just not their core interest. They certainly don't get taught anything of the sort, and so that's what I've been filling in for the past few years in a training capacity, but also in a consulting capacity. So quite often training people within an organisation and sometimes, for example, I'll get an engineering department of fifteen people and I'll force them to go and map out their relationships with other departments. They'll go and build relationships they've never done before and suddenly find new opportunities in their engineering world and they're amazed that having a chat with somebody would unlock opportunities. But it came back to The Virtual Flute - I'd had to do this because I'd forced myself to step out of computer science. Once you see the value of putting this discipline with that discipline, suddenly there's a new opportunity... It's not what we often think of as communication, like for example giving a speech - but it is highly communicative and highly collaborative. Once at the end of the flute project Joe and I - Joe Wolfe was my, sort of, primary supervisor on my project - we'd have discussions saying, "Who do we think should mark this project?" Because, again, he didn't have all the knowledge of the computing of it all so I went and looked for people to mark my project and, you know, what undergraduate does that? It's always the other way around. So I would knock on doors saying, "I'm so and so, I've been doing this work across the road in physics, what do you think?" And because I'm asking them to do something, I'd have to persuade them to come on board and be interested in the project. And so my current skills and interest in communication started from day one right there.