Hi nicolant,
This will be caused by the Zend_Search_Lucene indexer. What happens is, while it's building the index, it creates 'segment' files which are parts of the index and combines these when they pass a set threshold. You can actually change that threshold using the MergeFactor property of the index when you create it.
There's a mechanism in place in the plugin to let you change this thankfully. Try this code in your _config.php, you need to rebuild your index afterwards to make it take effect:
function create_index_callback() {
$index = ZendSeachLuceneWrapper::getIndex();
$index->setMergeFactor(5);
}
ZendSearchLuceneWrapper::addCreateIndexCallback('create_index_callback');
Have a read of this page and a play with MergeFactor values (default is 10, you want to set it lower) and you should be able to get your index working OK =]
http://framework.zend.com/manual/en/zend.search.lucene.index-creation.html
HTH!
Darren