[alicebot-archcomm] performance testing

[Noel Bush]

Hmmm.

> Idea 1:
> Post the URL of the bot to be tested. Ask everyone in the Alicebot
> community (via the mailing lists) to connect to the test bot in a
> specific time frame, for example 18:00-18:30 GMT. Each client should
> chat in such a way so as to not simply trigger the confusion statement
> all the time.

What I want to test is the performance of the bot on different machines.
Sorry I didn't make that clear.  It is not a problem to feed giant sets
of inputs to a bot; the problem is how to distribute a uniform set of
inputs to various testers, all of whom are running the bot on different
machines, so that the set of inputs doesn't raise privacy objections
because it came from real conversations.  (More below)

> Idea 2:
> Write some javascript into your test bot's template.html that parses
> sentences at random out of a large text file of a children's book or a
> Hemingway novel and submits them to the botserver after a few seconds.
> Each person who connects to this bot can just sit back as 
> their browser
> has a conversation with the bot. So you can open a dozen instances in
> your browser, or you can ask all 5 million people on the
> alicebot-general list to connect. When the testing is done, 
> take out the
> javascript and the cycle ends on the next refresh.

Problem is that the inputs ought to be real conversational inputs.  Text
from a novel won't produce meaningful results, I'm afraid, unless many
novels have been talking to ALICE over the last six years.

Also, to further clarify: the goal here is to test for performance
degradation during the matching process.  I think I have probably
"plugged" a lot of memory leaks, and have also sped up the matching
process, but it's necessary to find out if there are still problems that
cause matching to slow down over time or memory usage to rise to
unacceptable levels.  The issue of multiple clients, as I see it, is
wholly separate from this (and just as likely to be a problem of the web
server du jour as the matching engine).

> Idea 3: 
> Make two bots talk to each other over AOLIM.

This has the most interesting promise, although I think it's not
necessary to involve the IM interface.  It's possible to set up two bots
to talk to each other directly through the system shell, I think.  Some
preparation of a special AIML set might be in order here -- perhaps a
quick initial run would produce a set of target data that could be
mapped back to standard AIML to produce meaningless (but
engine-intensive) replies that could prevent the bots from quickly
devolving into a catchall festival (as I think would happen in the
unprepared case).