This add-on is operated by Yahoo Inc.

Search the web with Yahoo BOSS API

Last updated June 10, 2019

The Yahoo BOSS add-on provides access to the Yahoo BOSS Search API for your application.

Build web-scale search products utilizing the power of Yahoo Search technology and data. BOSS Search provides a rich set of premium data APIs and tools that developers and entrepreneurs can use to build custom search engines and innovative experiences.

Yahoo BOSS is accessible via a REST API that can be accessed from Node.js, Ruby, PHP, Python, Java, Clojure, Scala, and Groovy.

Provisioning the add-on

Yahoo BOSS add-on can be attached to a Heroku application via the CLI:

$ heroku addons:create yahooboss
-----> Adding yahooboss to sharp-mountain-4005... done, v18 (free)

Once Yahoo BOSS has been added, three config vars become available in the app configuration:

1. YAHOOBOSS_KEY - your unique access key
2. YAHOOBOSS_SECRET - the secret used to encrypt the key
3. YAHOOBOSS_URL - the API endpoint

These can be confirmed using the heroku config:get command.

$ heroku config:get YAHOOBOSS_URL
https://yboss.yahooapis.com/external/v1/web

After installing the add-on, your application should be configured to fully integrate with Yahoo BOSS.

Usage

Yahoo BOSS provides web search through a single HTTP GET call. To use it, you need to include a token in your HTTP header. Here are some code samples that show how to construct the header, using the config vars populated by the add-on.

API parameters

You can read the full web API documentation on the BOSS API page. Please note that at this point in time, we only offer support for the web endpoint. Here are some of the common parameters:

• q - search term. Cannot be empty. Can be a list of comma-separated terms.
• sites - restricts BOSS API search results to a set of pre-defined sites. Multiple sites must be comma-separated.
• format - can be json (default if omitted) or xml
• count - limit the number of results returned (1-50, default is 50).
• start - page through the results, by specifying the (0-based) result number to start from.

Expected response

For q=sunnyvale, the response would be:

{
  "bossresponse":{
    "responsecode":"200",
    "web":{
      "start":"0",
      "count":"50",
      "totalresults":"4570000",
      "results":[
        {
          "date":"",
          "clickurl":"http:\/\/sunnyvale.ca.gov\/",
          "url":"http:\/\/sunnyvale.ca.gov\/",
          "dispurl":"<b>sunnyvale<\/b>.ca.gov",
          "title":"City of <b>Sunnyvale<\/b>: Home",
          "abstract":"Official municipal site includes information about city services, departments, administration, meetings, events, and the Silicon Valley community."
        },
        {
          "date":"",
          "clickurl":"http:\/\/en.wikipedia.org\/wiki\/Sunnyvale,_California",
          "url":"http:\/\/en.wikipedia.org\/wiki\/Sunnyvale,_California",
          "dispurl":"en.wikipedia.org\/wiki\/<b>Sunnyvale<\/b>,_California",
          "title":"<b>Sunnyvale<\/b>, California - Wikipedia, the free encyclopedia",
          "abstract":"<b>Sunnyvale<\/b>, officially the City of <b>Sunnyvale<\/b>, is a city located in Santa Clara County, California. As of the 2010 United States Census, the population was ..."
        },
        "48 more results..."
      ]
    }
  }
}

Using with Node.js

//construct token, and 'X-Boss-Auth' header
var constructHeader = function() {
  var crypto = require('crypto');
  var util = require('util');
  var timestamp = parseInt((new Date()).getTime() / 1000, 10);
  var hmac = crypto.createHmac('sha1', process.env.YAHOOBOSS_SECRET).update(process.env.YAHOOBOSS_KEY).digest('hex');
  var token = crypto.createHash('md5').update(hmac + timestamp).digest('hex');
  var header = util.format('token id="%s", token="%s", ts="%s"', process.env.YAHOOBOSS_KEY, token, timestamp);
  return header;
};

//call API, and handle results/errors
var request = require('request');
var query = 'sunnyvale';
var options = {
  uri: process.env.YAHOOBOSS_URL + '?q=' + query,  //<--- add more parameters here
  method: 'GET',
  headers: {
    'X-Boss-Auth': constructHeader()
  }
};
request(options, function(error, response, body) {
  if(error) {
    console.error(error);
    //handle error
  }
  else {
    if(response.statusCode !== 200) {
      console.error(body);
      //handle error
    }
    else {
      //handle results
    }
  }
});

Using with PHP

<?php
  //construct token, and 'X-Boss-Auth' header
  $timestamp = intval(time(), 10);
  $hmac = hash_hmac('sha1', $_ENV['YAHOOBOSS_KEY'], $_ENV['YAHOOBOSS_SECRET']);
  $token = md5($hmac.$timestamp);
  $header = "token id=\"{$_ENV['YAHOOBOSS_KEY']}\", token=\"$token\", ts=\"$timestamp\"";

  //call API, and handle results/errors
  $query = 'sunnyvale';
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, $_ENV['YAHOOBOSS_URL'].'?q='.$query);
  curl_setopt($ch, CURLOPT_HTTPHEADER, array('X-Boss-Auth: '.$header));
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $output = curl_exec($ch);  //<-- handle output and errors
  curl_close($ch);
?>

Using with Ruby

require 'digest/md5'
require 'net/http'
require 'openssl'

YAHOOBOSS_KEY = ENV['YAHOOBOSS_KEY']
YAHOOBOSS_SECRET = ENV['YAHOOBOSS_SECRET']
YAHOOBOSS_URL = ENV['YAHOOBOSS_URL']

query = "sunnyvale"

digest = OpenSSL::Digest.new('sha1')
timestamp = Time.now.to_i.to_s
hmac = OpenSSL::HMAC.hexdigest(digest, YAHOOBOSS_SECRET, YAHOOBOSS_KEY)
token = Digest::MD5.hexdigest("#{hmac}#{timestamp}")

header = "token id=\"#{YAHOOBOSS_KEY}\", token=\"#{token}\", ts=\"#{timestamp}\""

uri = URI.parse(YAHOOBOSS_URL + "?q=" + query)
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
request = Net::HTTP::Get.new(uri.request_uri)
request.add_field('X-Boss-Auth', header)
puts uri.request_uri

response = http.request(request)
puts response.body

Using with Python

from __future__ import print_function
import hashlib
import hmac
import json
import os
import sys
import time

try:
  from urllib.parse import urlencode, urljoin
  from urllib.request import urlopen, Request
except ImportError:
  from urlparse import urljoin
  from urllib import urlencode
  from urllib2 import urlopen, Request


def search_boss(query):
  """
  Send a query to the Yahoo Boss service and get back a result.
  :param query: str to Query for
  :return: boss response dictionary
  """
  hmac_digest = bytes(
    hmac.new(
      key=bytes(os.environ['YAHOOBOSS_SECRET'].encode('ascii')),
      msg=bytes(os.environ['YAHOOBOSS_KEY'].encode('ascii')),
      digestmod=hashlib.sha1
    ).hexdigest().encode('ascii')
  )
  timestamp = bytes(int(time.time()))
  token = hashlib.md5(hmac_digest + timestamp).hexdigest()

  header = 'token id="{YAHOOBOSS_KEY}", token="{token}", ts="{timestamp}"'.format(
    YAHOOBOSS_KEY=os.environ['YAHOOBOSS_KEY'],
    token=token,
    timestamp=timestamp,
  )
  query_url = urljoin(os.environ['YAHOOBOSS_URL'], '?' + urlencode({'q': query}))

  request = Request(query_url)
  request.add_header('X-Boss-Auth', header)

  response = urlopen(request)
  data = response.read()

print(search_boss('sunnyvale'))

Using with Java

package com.yahoo.boss;

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import javax.net.ssl.HttpsURLConnection;
import java.io.*;
import java.net.MalformedURLException;
import java.net.URL;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.security.SignatureException;

public class BossDemo {
  private static final String HMAC_SHA1_ALGORITHM = "HmacSHA1";
  private static final String MD5_ALGORITHM = "MD5";

  private final String YAHOOBOSS_KEY = System.getenv("YAHOOBOSS_KEY");
  private final String YAHOOBOSS_SECRET = System.getenv("YAHOOBOSS_SECRET");
  private final String YAHOOBOSS_URL = System.getenv("YAHOOBOSS_URL");

  // Get the current timestamp in seconds as a string
  public static String getTimestamp() {
    return String.valueOf(System.currentTimeMillis() / 1000L);
  }

  // Utility method to convert bytes to a hex string
  public static String bytesToHexString(byte[] bytes){
    StringBuilder sb = new StringBuilder();
    for(byte b : bytes){
      sb.append(String.format("%02x", b&0xff)); }
    return sb.toString();
  }

  // Calculate the HMAC SHA1 signature given a secret and a key
  public static String calculateHMAC(String data, String secret)
    throws java.security.SignatureException
  {
    String result;
    try {
      SecretKeySpec signingKey = new SecretKeySpec(secret.getBytes(), HMAC_SHA1_ALGORITHM);
      Mac mac = Mac.getInstance(HMAC_SHA1_ALGORITHM);
      mac.init(signingKey);
      byte[] hmacBytes = mac.doFinal(data.getBytes());

      result = bytesToHexString(hmacBytes);

    } catch (Exception e) {
      throw new SignatureException("Failed to generate HMAC : " + e.getMessage());
    }
    return result;
  }

  // Generate the MD5 Hash for the provided string
  private static String generateMD5Hash(String input) {
    String result = "";
    try {
      MessageDigest md = MessageDigest.getInstance(MD5_ALGORITHM);
      byte[] digest = md.digest(input.getBytes("UTF-8"));
      result = bytesToHexString(digest);
    } catch (NoSuchAlgorithmException e) {
      e.printStackTrace();
    } catch (UnsupportedEncodingException e) {
      e.printStackTrace();
    }

    return result;
  }

  // Generate the X-Boss-Auth header from the YAHOOBOSS_KEY and YAHOOBOSS_SECRET
  private String generateHeader() {
    try {
      String ts = getTimestamp();  // Current time in seconds
      String hmac = calculateHMAC(YAHOOBOSS_KEY, YAHOOBOSS_SECRET);
      String token = generateMD5Hash(hmac + ts);  // HMAC appended with the current timestamp, converted to MD5

      // Format as the header [token id="<id>", token="<token>", ts="<timestamp>"]
      return String.format("token id=\"%s\", token=\"%s\", ts=\"%s\"", YAHOOBOSS_KEY, token, ts);
    } catch (SignatureException e) {
      e.printStackTrace();
      return "";
    }
  }

  // Perform the actual search given the query term "query"
  public String search(String query) {
    StringBuilder sb = new StringBuilder();

    try {
      String authHeader = generateHeader();
      String urlString = YAHOOBOSS_URL + "?q=" + query;   // <-- Add more parameters here

      System.out.println("Using X-Boss-Auth: " + authHeader);
      System.out.println("URL: " + urlString);

      URL url = new URL(urlString);
      HttpsURLConnection conn = (HttpsURLConnection)url.openConnection();
      conn.setRequestProperty("X-Boss-Auth", authHeader);
      InputStream ips = conn.getInputStream();
      InputStreamReader isr = new InputStreamReader(ips);
      BufferedReader in = new BufferedReader(isr);

      // Read JSON string results and accumulate them in the StringBuilder
      String inputLine;
      while ((inputLine = in.readLine()) != null)
        {
          sb.append(inputLine);
        }

      in.close();
    } catch (MalformedURLException urle) {
      urle.printStackTrace();
    } catch (IOException ioe) {
      ioe.printStackTrace();
    }

    // Return JSON result blob or an empty string if there was an error
    return sb.toString();
  }

}

Using with Scala

package com.yahoo.boss;

import javax.crypto.Mac
import javax.crypto.spec.SecretKeySpec
import javax.net.ssl.HttpsURLConnection
import java.io._
import java.net.MalformedURLException
import java.net.URL
import java.security.MessageDigest
import java.security.NoSuchAlgorithmException
import java.security.SignatureException
import scala.io.Source

class BossDemo {
  val HMAC_SHA1_ALGORITHM = "HmacSHA1"
  val MD5_ALGORITHM = "MD5"

  val YAHOOBOSS_KEY = System.getenv("YAHOOBOSS_KEY")
  val YAHOOBOSS_SECRET = System.getenv("YAHOOBOSS_SECRET")
  val YAHOOBOSS_URL = System.getenv("YAHOOBOSS_URL")

  // Get the current timestamp in seconds as a string
  def timestamp = s"${System.currentTimeMillis() / 1000L}"

  // Utility method to convert bytes to a hex string
  def bytesToHexString(bytes: Array[Byte]): String = {
    val sb = new StringBuilder()
    bytes.toList.foreach{b=>
      sb.append(f"$b%02x")
    }
    sb.toString()
  }

  // Calculate the HMAC SHA1 signature given a secret and a key
  def calculateHMAC(data: String, secret: String): String = {
    try {
      val signingKey = new SecretKeySpec(secret.getBytes(), HMAC_SHA1_ALGORITHM);
      val mac = Mac.getInstance(HMAC_SHA1_ALGORITHM)
      mac.init(signingKey)
      val hmacBytes = mac.doFinal(data.getBytes())

      bytesToHexString(hmacBytes)

    } catch {
      case e: Exception =>
        throw new SignatureException("Failed to generate HMAC : " + e.getMessage())
    }
  }

  // Generate the MD5 Hash for the provided string
  def generateMD5Hash(input: String): String = {
    try {
      val md = MessageDigest.getInstance(MD5_ALGORITHM)
      val digest = md.digest(input.getBytes("UTF-8"))

      bytesToHexString(digest)
    } catch {
      case e: NoSuchAlgorithmException => e.printStackTrace();""
      case e: UnsupportedEncodingException => e.printStackTrace();""
    }
  }

  // Generate the X-Boss-Auth header from the YAHOOBOSS_KEY and YAHOOBOSS_SECRET
  def generateHeader(): String = {
    try {
      val ts = timestamp  // Current time in seconds
      val hmac = calculateHMAC(YAHOOBOSS_KEY, YAHOOBOSS_SECRET)
      val token = generateMD5Hash(hmac + ts)  // HMAC appended with the current timestamp, converted to MD5

      // Format as the header [token id="<id>", token="<token>", ts="<timestamp>"]
      println("::token "+token)
      s"""token id=\"${YAHOOBOSS_KEY}\", token=\"${token}\", ts=\"${ts}\""""
    } catch {
      case e:SignatureException =>e.printStackTrace();""
    }
  }

  // Perform the actual search given the query term "query"
  def search(query: String): String = {
    // Return JSON result blob or an empty string if there was an error
     try {
      val authHeader = generateHeader();
      val urlString = YAHOOBOSS_URL + "?q=" + query   // <-- Add more parameters here

      println("Using X-Boss-Auth: " + authHeader)
      println("URL: " + urlString)

      val url = new URL(urlString)
      val conn = url.openConnection().asInstanceOf[HttpsURLConnection]
      conn.setRequestProperty("X-Boss-Auth", authHeader)
      val ips = conn.getInputStream()
      try {
          Source.createBufferedSource(ips).mkString
      } finally {
          ips.close()
      }
    } catch {
      case urle: MalformedURLException => urle.printStackTrace();""
      case ioe:  IOException => ioe.printStackTrace();""
    }
  }
}

Using with Clojure

(ns yahooboss
  (:require [clj-http.client :as client])
  (:import [javax.crypto Cipher Mac]
           [javax.crypto.spec SecretKeySpec IvParameterSpec]
           [java.security MessageDigest]))

(defn- bytes->hex
  [b]
  (apply str (map #(format "%02x" %) b)))

(defn- hmac
  "Generates a Base64 HMAC with the supplied secret on a string of data."
  [data secret]
  (let  [algo "HmacSHA1"
         mac  (Mac/getInstance algo)]
    (.init mac (SecretKeySpec. (.getBytes secret) algo))
    (bytes->hex (.doFinal mac (.getBytes data)))))

(defn- md5
  "md5 hex digest"
  [s]
  (let [md (MessageDigest/getInstance "MD5")]
    (bytes->hex (.digest md (.getBytes s "UTF-8")))))

(defn- boss-header
  []
  (let [key (System/getenv "YAHOOBOSS_KEY")
        ts (str (long (/ (System/currentTimeMillis) 1000)))
        token (md5 (str (hmac key (System/getenv "YAHOOBOSS_SECRET")) ts))]
    (format "token id=\"%s\", token=\"%s\", ts=\"%s\"" key token ts)))

(defn search
  "Search using yahooboss."
  ([query]
   (search query 0 50))
  ([query start count]
   (let [resp (client/get (str (System/getenv "YAHOOBOSS_URL") "?format=json&q=" query "&start=" start "&count=" count)
                          {:headers {"X-Boss-Auth" (boss-header)}
                           :as :auto})]
     (:body resp))))

(defn- lazy-search-seq
  [query start count]
  (lazy-seq (let [r (search query start count)]
              (concat (seq (get-in r [:bossresponse :web :results]))
                      (lazy-search-seq query (+ count (Integer/parseInt (get-in r [:bossresponse :web :start]))) count)))))

(defn search-seq
  "lazy sequence of search results"
  [query]
  (lazy-search-seq query 0 50))

(comment
 (search "sunnyvale")

 (map :abstract (take 7 (search-seq "sunnyvale"))))

Using with Groovy

import groovy.transform.Memoized

import java.security.MessageDigest

import javax.crypto.Mac
import javax.crypto.spec.SecretKeySpec

public class BossDemo {
    private static final String YAHOOBOSS_KEY = System.getenv("YAHOOBOSS_KEY")
    private static final String YAHOOBOSS_SECRET = System.getenv("YAHOOBOSS_SECRET")
    private static final String YAHOOBOSS_URL = System.getenv("YAHOOBOSS_URL")

    // Calculate the HMAC SHA1 signature given a secret and a key
    @Memoized
    private static String calculateHMAC(String data, String secret) {
        Mac.getInstance('HmacSHA1').with {
            init(new SecretKeySpec(secret.bytes, 'HmacSHA1'))
            doFinal(data.bytes).encodeHex().toString()
        }
    }

    // Generate the MD5 Hash for the provided string
    private static String generateMD5Hash(String input) {
        byte[] hash = MessageDigest.getInstance('MD5').digest(input.bytes)
        new BigInteger(1, hash).toString(16).padLeft(32, '0')
    }

    // Generate the X-Boss-Auth header from the key and secret
    private static String generateHeader(String key, String secret) {
        String ts = (System.currentTimeMillis() / 1000).toString()
        String token = generateMD5Hash(calculateHMAC(key, secret) + ts)
        "token id=$key, token=$token, ts=$ts"
    }

    // Perform the actual search given the query term "query"
    public static String search(String query) {
        String urlString = "$YAHOOBOSS_URL?q=$query" // <-- Add more parameters here
        urlString.toURL().getText(requestProperties: ['X-Boss-Auth': generateHeader(YAHOOBOSS_KEY, YAHOOBOSS_SECRET)])
    }
}

println BossDemo.search("Yahoo")

Troubleshooting

Here are some common HTTP errors and the way to handle them:

Removing the add-on

Yahoo BOSS can be removed via the CLI.

$ heroku addons:destroy yahooboss
-----> Removing yahooboss from sharp-mountain-4005... done, v20 (free)

Support

All Yahoo BOSS support and runtime issues should be submitted via one of the Heroku Support channels. Any non-support related issues or product feedback is welcome at BOSS Heroku Support.