SearchBox Elasticsearch

This add-on is operated by Sebula Bilisim Teknolojileri LTD STI

Search Made Simple.

SearchBox Elasticsearch

Last updated September 11, 2020

SearchBox is an add-on for providing full-text hosted search functionality powered by Elasticsearch.

SearchBox offers real time searching, bulk indexing, faceting, geo tagging, auto-complete, suggestions (did you mean support) , saved queries(percolation) and many more without headache.

Installing the add-on

SearchBox can be installed to a Heroku application via the CLI:

$ heroku addons:create searchbox

For specific version of Elasticsearch please use CLI command with version information as;

$ heroku addons:add searchbox:starter --es_version=2
  • Available versions are 0,1,2,5,6 and 7

Once Searchbox has been added SEARCHBOX_URL setting will be available in the app configuration and will contain the account name and credentials to access SearchBox indices service. This can be confirmed using the heroku config command.

$ heroku config | grep SEARCHBOX_URL
SEARCHBOX_URL  => https://paas:8ed0986ecaabcb7c20b4b2bdd6251f2d@.....searchly.com

After installing SearchBox the application should be configured to fully integrate with the add-on.

SearchBox does NOT creates index automatically. So ensure to create an index via API or dashboard.

Using with Rails

Elasticsearch Rails client is a Ruby client for the Elasticsearch supports;

  • ActiveModel integration with adapters for ActiveRecord and Mongoid
  • Repository pattern based persistence layer for Ruby objects
  • Active Record pattern based persistence layer for Ruby models
  • Enumerable-based wrapper for search results
  • ActiveRecord::Relation-based wrapper for returning search results as records
  • Convenience model methods such as search, mapping, import, etc
  • Rake tasks for importing the data
  • Support for Kaminari and WillPaginate pagination
  • Integration with Rails’ instrumentation framework
  • Templates for generating example Rails application

A sample Rails application can be found on GitHub https://github.com/searchly/searchly-rails-sample.

Configuration

Ruby on Rails applications will need to add the following entry into their Gemfile.

gem 'elasticsearch-model'
gem 'elasticsearch-rails'

Update application dependencies with bundler. term $ bundle install Configure Rails Elasticsearch in configure/application.rb or configure/environment/production.rb

Elasticsearch::Model.client = Elasticsearch::Client.new host: ENV['SEARCHBOX_URL']

if given connection URL does not working, try setting https default port explicitly as;

Elasticsearch::Model.client = Elasticsearch::Client.new(host: ENV['SEARCHBOX_URL'], http: { port: 443, scheme: 'https' })

Index Creation

First add required mixin to your model;

class Document < ActiveRecord::Base
  include Elasticsearch::Model
end

From Rails console, create documents index for model Document.

Document.__elasticsearch__.create_index! force: true

Make your model searchable:

class Document < ActiveRecord::Base
  include Elasticsearch::Model
  include Elasticsearch::Model::Callbacks
end

When you now save a record:

Document.create :name => "Cost",
               :text => "Cost is claimed to be reduced and in a public cloud delivery model capital expenditure is converted."

The included callbacks automatically add the document to a documents index, making the record searchable:

@documents = Document.search('Cost').records

Elasticsearch Rails has very detailed documentation at official Elasticsearch page.

Using Searchly with Python

Elasticsearch provides two types of clients for Python low-level and elasticsearch-dsl.

Using with Python

Low-level client

Compatibility

The library is compatible with all Elasticsearch versions since 0.90.x but you have to use a matching major version:

Set your requirements in your setup.py or requirements.txt is:

# Elasticsearch 7.x
elasticsearch>=7.0.0,<8.0.0
# Elasticsearch 6.x
elasticsearch >= 6.0.0,<7.0.0
# Elasticsearch 5.x
elasticsearch >= 5.0.0,<6.0.0
# Elasticsearch 2.x
elasticsearch> = 2.0.0,<3.0.0

Install via pip;

$ pip install elasticsearch

Connection & Authentication

from elasticsearch import Elasticsearch

from urllib.parse import urlparse
# for python2;
# from urlparse import urlparse

url = urlparse(os.environ.get('SEARCHBOX_URL'))

es = Elasticsearch(
    [url.host],
    http_auth=(url.username, url.password),
    scheme=url.scheme,
    port=url.port,
)

Example Usage

from datetime import datetime
from elasticsearch import Elasticsearch
from urlparse import urlparse

url = urlparse(os.environ.get('SEARCHBOX_URL'))

es = Elasticsearch(
    [url.host],
    http_auth=(url.username, url.password),
    scheme=url.scheme,
    port=url.port,
)

doc = {
    'author': 'kimchy',
    'text': 'Elasticsearch: cool. bonsai cool.'
}

# ignore 400 cause by IndexAlreadyExistsException when creating an index
es.indices.create(index='test-index', ignore=400)

# create tweet
res = es.index(index="test-index", id=1, body=doc)
print(res['created'])

# get tweet
res = es.get(index="test-index", id=1)
print(res['_source'])

# search tweet
res = es.search(index="test-index", body={"query": {"match_all": {}}})
print("Got %d Hits:" % res['hits']['total'])
for hit in res['hits']['hits']:
    print("%(author)s: %(text)s" % hit["_source"])

DSL (high-level client)

The library is compatible with all Elasticsearch versions since 1.x but you have to use a matching major version:

# Elasticsearch 7.x
elasticsearch-dsl>=7.0.0,<8.0.0
# Elasticsearch 6.x
elasticsearch-dsl>= 6.0.0,<7.0.0
# Elasticsearch 5.x
elasticsearch-dsl>= 5.0.0,<6.0.0
# Elasticsearch 2.x
elasticsearch-dsl>= 2.0.0,<3.0.0
# Elasticsearch 1.x
elasticsearch-dsl<2.0.0

Connection & Authentication

You can either manually pass your connection instance to each request or set default connection for all requests.

Default connection

To define a default connection that will be used globally, use the connections module and the create_connection method:

from elasticsearch_dsl import connections

connections.create_connection(hosts=[os.environ.get('SEARCHBOX_URL')], timeout=20)

Indexing

You can create model-like wrapper for your documents via using DocType class. It can provide mapping and settings for Elasticsearch.

from elasticsearch_dsl import DocType, Date, Nested, Boolean,
    analyzer, InnerDoc, Completion, Keyword, Text

html_strip = analyzer('html_strip',
    tokenizer="standard",
    filter=["standard", "lowercase", "stop", "snowball"],
)

class Comment(InnerDoc):
    author = Text(fields={'raw': Keyword()})
    content = Text(analyzer='snowball')

class Post(DocType):
    title = Text()
    title_suggest = Completion()
    published = Boolean()
    category = Text(
        analyzer=html_strip,
        fields={'raw': Keyword()}
    )

    comments = Nested(Comment)

    class Meta:
        index = 'blog'

    def add_comment(self, author, content):
        self.comments.append(
          Comment(author=author, content=content))

    def save(self, ** kwargs):
        return super().save(** kwargs)

Also you can explicitly create index and provide settings at index;

from elasticsearch_dsl import Index, DocType, Text, analyzer

blogs = Index('blogs')

# define aliases
blogs.aliases(
    old_blogs={}
)

# register a doc\_type with the index
blogs.doc_type(Post)

# can also be used as class decorator when defining the DocType
@blogs.doc_type
class Post(DocType):
    title = Text()

# You can attach custom analyzers to the index

html_strip = analyzer('html_strip',
    tokenizer="standard",
    filter=["standard", "lowercase", "stop", "snowball"],
)

blogs.analyzer(html_strip)

# delete the index, ignore if it doesn't exist
blogs.delete(ignore=404)

# create the index in elasticsearch
blogs.create()

Create and save documents to Elasticsearch;

# instantiate the document
first = Post(title='My First Blog Post, yay!', published=True)
# assign some field values, can be values or lists of values
first.category = ['everything', 'nothing]
# every document has an id in meta

first.meta.id = 47

# save the document into the cluster
first.save()

### Searching

# by calling .search we get back a standard Search object
s = Post.search()
# the search is already limited to the index of our document
s = s.filter('term', published=True).query('match', title='first')

results = s.execute()

# when you execute the search the results are wrapped in your document class (Post)
for post in results:
    print(post.meta.score, post.title)

Using with Node.js

Elasticsearch Node.js client is official client for Node.js.

A sample Node.js application can be found on GitHub https://github.com/searchly/searchly-nodejs-sample.

Configuration

Add elasticsearch dependency to your package.json file and use npm to install your dependencies

"dependencies": {
   "elasticsearch": ">=1.1.0"
}

Create a search client:

var elasticsearch = require('elasticsearch');

var connectionString = process.env.SEARCHBOX_URL;

var client = new elasticsearch.Client({
    host: connectionString
});

Index a document

client.index({
  index: 'sample',
  type: 'document',
  id: '1',
  body: {
          name: 'Reliability',
          text: 'Reliability is improved if multiple redundant sites are used, which makes well-designed cloud computing suitable for business continuity.'
  }
}, function (error, response) {
  console.log(response);
});

Create a query and search it

client.search({
        index: 'sample',
        type: 'document',
        body: {
            query: {
                query_string:{
                   query:"Reliability"
                }
            }
        }
    }).then(function (resp) {
        console.log(resp);
    }, function (err) {
        console.log(err.message);
    });

Detailed documentation for Nodejs client can be found here

Using With Java

Using Elasticsearch Java High Level REST Client

Installation

Add dependency to your pom.xml (Ensure client and Elasticsearch versions are compatible);

<dependency>
    <groupid>org.elasticsearch.client</groupid>
    <artifactid>elasticsearch-rest-high-level-client</artifactid>
    <version>7.3.0</version>
</dependency>

Update application dependencies with maven;

$mvn clean install

or gradle;

compile group: 'org.elasticsearch.client', name: 'elasticsearch-rest-high-level-client', version: '7.3.0'

./gradlew install

Initialize Client

RestHighLevelClient client = null;
try {
    URI uri = new URI(System.getenv("SEARCHBOX_URL"));

    final String[] userInfo = uri.getUserInfo().split(":");
    final CredentialsProvider credentialsProvider =
            new BasicCredentialsProvider();
    credentialsProvider.setCredentials(AuthScope.ANY,
            new UsernamePasswordCredentials(userInfo[0], userInfo[1]));

    client = new RestHighLevelClient(
            RestClient.builder(
                    new HttpHost(uri.getHost(), uri.getPort(), uri.getScheme()))
                    .setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
                        @Override
                        public HttpAsyncClientBuilder customizeHttpClient(
                                HttpAsyncClientBuilder httpClientBuilder) {
                            return httpClientBuilder
                                    .setDefaultCredentialsProvider(credentialsProvider);
                        }
                    }));

    // put actions here
    // CreateIndexRequest request = new CreateIndexRequest("articles");
    // CreateIndexResponse createIndexResponse = client.indices().create(request);
    //.
    //.

} catch (IOException | URISyntaxException e) {
    e.printStackTrace();
} finally {
    if (client != null)
        try {
            client.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
}

Index Creation

CreateIndexRequest request = new CreateIndexRequest("articles");
CreateIndexResponse createIndexResponse = client.indices().create(request);

Search

Index an article;

IndexRequest request = new IndexRequest(
        "articles",
        "doc",
        "1");
String jsonString = "{" +
        "\\"author\\":\\"john ronald reuel tolkien\\"," +
        "\\"content\\":\\"the lord of the rings is an epic high fantasy novel\\"," +
        "}";
request.source(jsonString, XContentType.JSON);
IndexResponse indexResponse = client.index(request);

You can search indexed article as:

SearchRequest searchRequest = new SearchRequest();
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.query(new MatchQueryBuilder("content", "lord"));
searchRequest.source(searchSourceBuilder);

SearchResponse searchResponse = client.search(searchRequest);

Useful Resources

Java Rest client has very detailed documentation at at Github.

Using Jest for Elasticsearch version below 6

A sample Java application can be found on GitHub https://github.com/searchly/searchly-java-sample.

Installation

Documentation targets Java client Jest for Elasticsearch.

Add Jest dependency to your pom.xml (Ensure client and Elasticsearch versions are compatible);

<dependency>
    <groupid>io.searchbox</groupid>
    <artifactid>Jest</artifactid>
    <version>0.1.6</version>
</dependency>

Update application dependencies with maven;

$ mvn clean install

Create Client

// construct a new jest client according to configuration via factory
 Jestclientfactory factory = new Jestclientfactory();
 factory.sethttpclientconfig(new HttpClientConfig
                        .builder(System.getenv("SEARCHBOX_URL"))
                        .multithreaded(true)
                        .build());
 jestclient client = factory.getobject();

Index Creation

client.execute(new CreateIndex.Builder("articles").build());

Search

Index an article;

Article source = new Article();
source.setauthor("john ronald reuel tolkien");
source.setcontent("the lord of the rings is an epic high fantasy novel");

Index index = new Index.Builder(source).index("articles").type("article").build();
client.execute(index);

You can search indexed article as:

String query = "{ \\"query\\": { \\"match\\" : { \\"content\\" : \\"lord\\" } } }";
Search search = (search) new search.builder(query)
                .addindex("articles")
                .addtype("article")
                .build();
Jestresult result = client.execute(search);

Useful Resources

Jest client has very detailed documentation at at Github.

SearchBox Elasticsearch dashboard

The SearchBox dashboard allows you to create, delete and edit access configurations of your indices and also gives basic statistical information.

The dashboard can be accessed via the CLI:

$ heroku addons:open searchbox
Opening searchbox for sharp-mountain-4005…

or by visiting the Heroku Dashboard and selecting the application in question. Select SearchBox from the Add-ons menu.

Migrating between plans

Use the heroku addons:upgrade command to migrate to a new plan.

$ heroku addons:upgrade searchbox:basic
-----> Upgrading searchbox:basic to sharp-mountain-4005... done
Your plan has been updated to: searchbox:basic

SEARCHBOX_URL and SEARCHBOX_SSL_URL are set to your Heroku application’s env variable and can be changed via SearchBox dashboard.

From Heroku dashboard -> resources click SearchBox addon and at SearchBox dashboard navigate to ACL->API Key. Click Generate New Api Key will generate new API Key and will update SEARCHBOX_URL and SEARCHBOX_SSL_URL variables at your Heroku application.

Removing the add-on

SearchBox can be removed via the CLI.

This will destroy all associated data and cannot be undone!

$ heroku addons:destroy searchbox
-----> Removing searchbox from sharp-mountain-4005... done, v20 (free)

Troubleshooting

  • SearchBox uses secure connection and port 443. If your client returning connection refused exception, please ensure you are using port 443 instead of default port of Elasticsearch which is 9200.

API limitations

Index refresh times and shard/replica counts can not be modified.

Additionally, all administrative features of ElasticSearch are restricted from the API.

Support

All SearchBox support and runtime issues should be submitted via on of the Heroku Support channels. Any non-support related issues or product feedback is welcome at SeachBox Support.

Additional resources

Feedback

Log in to submit feedback.