Thanks for an interesting talk about this topic at eurocon. Integrating the pipeline with UpdateRequestProcessorChain as Jan mentions is actually what I was thinking about talking to you after the talk. Also for simple needs – why not only implement a (few) stage/functionality directly as a stage in this chain (sub-classing UpdateRequestProcessorFactory) and configuring it in (a possibly seperate chain in) solrconfig.xml?

I looked at commons pipeline, and what it does particularly well is scalability in terms of the individual processing stages – each stage can have its own thread! – and queuing between stages etc. Also the parallell nature allows for optimized utilization for heavy processing, although for many search use cases order is important so I don’t know about the usefulness of this.

Carl M: I did not know about SMILA. Impressive. However, it feels way too heavy for what majority of Solr installations will be needing. I foresee SMILA useful only for the largest integrations where they are already using BPEL, not for the simple processing needs of a local search application. Also it smells a bit over-engineered?

Back to your suggestion Max. If the pipeline is to live strictly outside of Solr it makes sense to speak Solr-API in both ends. However, you then short-circuit all other UpdateRequestHandlers from benefiting from the pipeline. DIH, CSV, Extracting, (Java)Binary and other handlers cannot use it.

To overcome this, instead of integrating the pipeline as a standalone service, integrate it into Solr’s UpdateRequestProcessorChain, which sits between the RequestHandlers and indexing. There could be two versions of the pipeline factory – one Local, which executes the pipeline in same thread, and one Remote which streams the documents to a dedicated processing node/cluster.

I think this plays better with the current and future Solr architecture because
* The pipeline will be truly transparent, and ALL current RequestHandlers can be used, including DIH and LCF
* Solr will get built-in shard routing logic, obeying the new concepts of collections etc from SolrCloud
* It makes sense to have the choice of running a light-weight single-node without unnecessary HTTP calls, but also have the possibility of scaling out

Another issue is that the REST document model and SolrInputDocument is not (currently) rich enough to hold metadata about a partly processed document. I think it is unavoidable at some point to at least do Tokenization in the pipeline. Then we need to pass on a document with both the original version of the field as well as the tokenized version with metadata telling that it is already tokenized. The Lucene analysis chain could then skip tokenization. This is the way OpenPipe (http://openpipe.berlios.de/) is integrated with Solr. They found the need to invent a custom binary protocol to convey such metadata…

I really like the idea of using commons-pipeline, keep up the thinking

Currently under (heavy?) development by two companies from germany (Empolis, Brox).

- Categorization.
- Language detection.
- Different language processing depending on the language detection.
- And so much more.

When it comes to the DIH I´m no fan of it. I really have no reason for not using it, but it just seems complicated.

And yes, it’s another moving part of Solr, but (!) highly optional. This type of product would only be used for applications where you need to manipulate your data a lot. In for example large enterprise applications, wiki´s, library content etc. etc.

Pros:
– Speaking Solr at both ends makes it much easier to play with. One of the things I love about Solr is how simple it is to start using it. No need to set up crazy message queues, complex documents, etc. Just pop a document in. So this would lower the barrier to entry.
– Give a hook into what is going into Solr without changing Solr itself. I can see putting this as a frontend and just logging/recording what is going through. Most of us probably can not easily tell “what docs changed in the last 15 minutes” for example.
- Poor mans ESB… When I index a document of type X, also trigger this other job or something can go there.
- Maybe a simpler way to share processing steps by fitting into a standard pipeline?

Cons:
– Doesn’t this to some extent duplicate the index time pipeline inside of Solr for various document types?
– Another moving part for Solr?
– Are you duplicating a lot of DIH does in terms of processing logic? Although, I actually don’t really like the DIH approach from an approach. It seems like it solves a specific issue well, but then the more processing you do, the more you are writing code/logic using complex XML!

Something about your proposal sounded familiar, and I looked back at some projects and a couple of years ago built a data processing pipeline using Apache Commons Chain. An XML doc defined the pipeline and any specific values per step, and then you could have multiple pipelines. The ability to have multiple ones and try one versus the other was great. Love to see where this goes. Maybe replace DIH’s processing layer?

I know every time I write an indexer for a new project, I often put some of the same code over and over!

But I think that most people would have their own “hacked” connectors that speak “Solr-REST”. And in that case it would be nice to have a pipeline that acts as a Solr on both sides. A more or less totally transparent pipeline.

Let’s see what others say:
http://search-lucene.com/m/B1LSk1RL4BN