Search in Drupal 8
It has been largely agreed that core search in D8 should therefore be split into a generic framework providing backend- and data-independent tools for providing indexing and search capabilities, and a default implementation of that framework that small users can instantly use to add search capabilities to their sites.
Advanced search solutions like other backends (Solr, Xapian, Sphinx, …), facetted searches, searching of external content, etc., should easily be able to build on the core search framework to provide a unified API to all kinds of searches, and to avoid re-inventing the wheel over and over for all backends and purposes.
There are already several discussions and other resources detailing all aspects of that new search framework:
As a base solution I'd suggest the Search API module, which already supports most of the required features and could be adapted well enough to provide the others as well for D8 core.
I'd be willing to take a shot at that and would present a rough plan for it in this core conversation. Then, the discussion could work with a concrete proposal for which possible flaws, short-comings and open points could be debated.
Also it could be discussed how this will fit in with large other changes in D8, like i18n, configuration management, plugin system, entities, etc. It will especially be important if there will be something like the Entity API (or at least its property information system) in D8 core.
Search toolkit for Drupal 8
The current Search module suffers from a lack of maintainer-ship. We need to simplify the functionality provided by the core back-end so it becomes easier to maintain (and test) going forward.
In addition, new effort should focus on providing in Core a Search Toolkit that provides a framework and basic implementation for the UI, but moves away from any attempt to interface generically with the indexing process or other pieces that are back-end specific. A key portion of such a toolkit would be adapted from the Facet API module
In addition, we should remove the advanced search form from core, remove most of the complex query parsing from core, and provide implementations of a couple facets for core via the SQL back-end.
The net advantage is that new search UI elements could be used across back-ends, and theming would be more consistent regards of how the search results are generated.