New Developments in the Text-to-Speech Keyboard

Recently my brother, Nets Hawk Katz, sent me a new keyboard program for Bow to use on his touchscreen computer. He was learning a new programming language called GameMaker and so he used it to create a new program to put on Bow's touchscreen. Nets has made the different versions of the program available for download on his site. The source code is also available in case you are interested in modifying the program yourself, but to do so you would need to get GameMaker.

The first version of the program that Nets sent me was keyboard1.exe. I downloaded the program, installed it on Bow's computer, and then I said to Bow: "Uncle Nets has sent you a new program to use on your computer. Would you like to try it?"

Bow did not answer the question directly. Instead he spelled: "Uncle Nets is good."

I took this as a positive response, and I let Bow play with the program. Here is some footage of that:

This was on July 8. Though Bow was pleased to be able to play with the program, his use of it was not especially productive. I was not able to open it for him in full screen, so it was a small window superimposed on the desktop. Bow could close the window and open other programs, and this disrupted any communicative use of the program. He had fun playing with his computer, but it was not being used as intended. Other issues were that the delay between writing and then hearing what was written pronounced seemed too long.

Nets then sent us the next version of the program: Keyboard2.exe. The delay was changed, noises were made every time a key was pressed and a transcript file was added. Nets wrote: "The keyboard writes a transcript in a file transcript.txt in the same directory where you place the executable. (This transcript contains blank lines every time the timer runs out when nothing has been written.) You don't have to remove the transcript file every time you use the keyboard. The program simply appends additional lines to the end. "


I went and installed the new version of the program as soon as I received it, but Bow did not want to try it because it was late afternoon and nearly dinner time, and after dinner he was too content and exhausted to do anything more than lie around and chew his cud. Here is some footage of me playing with second version of the keyboard, without Bow:

Nets told me I was confusing "maximize" with "full screen." I would normally get full screen by pressing F4, but that was hard to do with a touchscreen and no keyboard. I had a keyboard I could attach, but it was not in the pen. There is also a virtual keyboard as part of the operating system of the touchscreen, but I had trouble finding it. Before I found another way to resolve this issue, Nets had already written another version of the keyboard to take care of it: Keyboard3.exe. It allows for an irrevocable full screen option. Once in full screen, no way back out, other than closing the window.

Here is some footage of our use of Keyboard3.

Under Keyboard3, here are some things that Bow typed:

IJ

EKFEKUP PO

UUU

NNNNNUZ

Arguably, the last one could have been a reference to Nets, as his name is spelled NZ in Hebrew. But that might be stretching things. Bow did not seem to be using the keyboard to communicate. He was just playing. 

The keyboard is for English words. Bow and I do not speak English with each other. Hebrew text to speech is very complicated, so this keyboard is meant to be used by Bow to communicate with English speakers, such as Lawrence. By the time Lawrence came over on Wednesday to spend the day with Bow, Nets had created a new version of the keyboard: Keyboard4.exe. There was no more x for closing the window. 

I was going to leave Bow and Lawrence with the keyboard, and I was hoping that any interactions would be recorded on the transcript, but I also went out and got a cheap camera from Wal*Mart for Lawrence to film Bow using the keyboard. The transcript shows everything that is typed, but it does not show who typed it. So I wanted footage of Bow typing and a transcript to show what he typed.

However, at the end of the day, Lawrence reported that Bow would not type anything when he, Lawrence, was in the room. But when Lawrence went to empty the potty, Bow took the opportunity to use the keyboard.

This morning I looked at the transcript. Here is what I found from yesterday:

B

O

BOW

BO

WHHAT

SAY

BO

DO

U

WA

N

TO

I'm pretty sure most of that was written by Lawrence. Was any part of it by Bow? I have to find a better protocol in order to distinguish what each contributed to the text. There's no video footage, because Lawrence was charging the new camera most of the day. Pretty inconclusive...

Meanwhile, now that I have a new camera, I tried taking some still shots of Bow.

This is Bow's evening face from last night, when he was chewing his cud after dinner. Here is a shot of Bow outside this morning:

Shortcuts in spelling

After Megan's departure, things settled down to their regular routine. Bow was satisfied. I was spending my regular twelve hours in the pen with him, and he gave no trouble, offered no resistance, but behaved like a perfect little gentleman.

There was a carnival in the small town closest to us, and Sword and I went there, two nights in a row. Each night, without making any appointment in advance, Sword spotted a friend, and the two of them would go on all the rides together, chattering away to each other as they spun around or were tossed about on a miniature roller coaster. Each time, I made conversation with the mother of Sword's friend, but the small talk we exchanged could not include: I have a little chimp boy at home who can spell and speak in complete sentences, but I just can't get him to behave. In both cases, the other mothers knew about Bow and even inquired about him, but the sorts of answers I gave were about the small stuff, and not about the fact that he has language.

Something really enormous has happened here, but because the breakthrough occurred three years ago, and got no acknowledgment, we tend to talk about other things. Somebody learns to talk or to spell only once in a lifetime, and after a while those things become routine, and we take no notice of them anymore.

On Sunday, we had a visit from a fellow primatologist. She was one of the professors who had recommended Megan, and she knew all about Megan's progress with Bow. Megan had emailed her telling of how at first Bow was reluctant to speak to her, then used single words, and finally ended up using full sentences with her. She even related how Bow had told her he was "lucky".  And then, of course, she also told of how Bow had been mean to her, and why she decided not to stay.

Bow liked the primatologist, and played chase with her through the glass. After she left, he said that she was good and that he hoped she would come again.

Then on Monday, Lawrence came, and there was a sort of mini-breakthrough on the touchscreen. For the  longest time, Bow hasn't said a single thing using his computer, preferring to break the chopsticks rather than point at letters. But yesterday, Lawrence was just sitting there with Bow, and he decided to play with the touchscreen himself. There was no plan or intent, involved. He just spelled "b-o-w", and the computer pronounced it wrong, as if it were a command to bow down to someone. (I've known for a long time that it mispronounces Bow's name, but there's not much I can do about it.) However, Lawrence wanted it to pronounce Bow's name correctly, so he spelled "b-o", and the computer said it right this time.

Bow had been watching lazily from the sidelines. But now he came over, took the chopstick from Lawrence and spelled "b-o". Again, the computer pronounced his name.

At this point, Bow lost interest in the computer and began playing roughly with the chopstick. Lawrence, concerned that he would break it, said:  "Bow, give me the chopstick!"

Bow looked at him in the manner of an impudent child, then he took the chopstick and pointed at a single letter: "y". The computer said: "Why?"

Lawrence is convinced that this is exactly what Bow meant to do: to challenge Lawrence verbally over his command to relinquish the chopstick.

Two issues spring to mind: if the computer allows for creative spelling, how do we prove what Bow meant? Secondly, do we really want a computer program that encourages Bow to spell things wrong? On the glass, he would never have thought to say "y". He would have spelled it out: "w-h-y".