Einstein Vision and Language

This add-on is operated by Salesforce

Build smarter apps with image recognition and natural language processing

Einstein Vision and Language

Last updated October 06, 2022

The Einstein Vision and Einstein Language APIs are part of the Salesforce Einstein suite of technologies, and you can use them to AI-enable your apps. Using the Einstein Vision APIs, you can leverage pre-trained classifiers or train your own custom classifiers to solve a vast array of specialized image recognition use cases. With the Einstein Language APIs, you can harness the power of natural language processing to analyze text and infer the sentiment or intent behind that text.

Provisioning the add-on

You can provision the Einstein Platform Services add-on for your app using the CLI:

$ heroku addons:create einstein-vision:starter --app coral-brook-39111
Creating einstein-vision:starter on ⬢ coral-brook-39111... free

A list of all plans available can be found here.

Einstein Platform Services authorization

After you install the add-on, these config vars are available:

  • EINSTEIN_VISION_ACCOUNT_ID - Your account ID.
  • EINSTEIN_VISION_PRIVATE_KEY - An RSA key in PEM format.
  • EINSTEIN_VISION_URL - The Einstein Vision and Lanuage APIs endpoint.

You can view these variable values with the heroku config:get command:

$ heroku config:get EINSTEIN_VISION_ACCOUNT_ID
heroku@famj5Uu91v6ByMC85f1aCfqJg1484200000000

$ heroku config:get EINSTEIN_VISION_PRIVATE_KEY
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEAze8Dh6o66ThPZz7 ...
...
-----END RSA PRIVATE KEY-----

$ heroku config:get EINSTEIN_VISION_URL
https://api.einstein.ai/

Authorization flow

The Einstein Platform Services APIs use OAuth 2.0 authorization. Each API call must pass in a valid OAuth access token.

In your app, you must add the code that exchanges a signed JWT (JSON web token) for the OAuth access token. To get started quickly, you can use our token generation page to generate a temporary token without code.

Steps to get a token

These are the high-level steps you follow in code to get a token. The code details vary depending on the language you’re using.

  1. Sign the JWT with your private key to create an assertion.
  2. Request an access token by sending the JWT assertion to the /v2/oauth2/token API.
  3. Use the access token in the Authorization request header to make API calls.
Example JWT payload:
{
  "sub": "$EINSTEIN_VISION_ACCOUNT_ID",
  "aud": "https://api.einstein.ai/v2/oauth2/token",
  "exp": $EXPIRY_UNIX_TIME_IN_SECONDS
}

exp: The expiration time in Unix time. This value is the current Unix time in seconds plus the number of seconds you want the token to be valid. In the JWT payload, you set the amount of time the token is valid, after which the token expires. You must generate a new token when the current token expires.

Using either a JWT library, custom code, or this token generation page, sign the JWT payload using the RSA private key config var EINSTEIN_VISION_PRIVATE_KEY to generate the JWT assertion.

Example JWT for access token exchange

Use the JWT assertion to generate a JWT token using the OAuth token generation resource https://api.einstein.ai/v2/oauth2/token.

$ curl -H "Content-type: application/x-www-form-urlencoded" \
    -X POST https://api.einstein.ai/v2/oauth2/token \
    -d "grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer&assertion=eyJhOiJSU ..."

{
    "access_token": "4f4c5a29a284920dcd43700dcc56c0ff2e2e2cb5",
    "token_type": "Bearer",
    "expires_in": 900
}

For more information, see the OAuth token API documentation.

You must set the token in the API request header to authenticate each API call. Use this header value: Authorization: Bearer <TOKEN>

Example API call

This cURL command returns a list of all your datasets.

$ curl -X GET \
    -H "Authorization: Bearer 4f4c5a29a284920dcd43700dcc56c0ff2e2e2cb5" \
    https://api.einstein.ai/v2/vision/datasets

{
  "object": "list",
  "data": [
    {
      "id": 24,
      "name": "Beach and Mountain",
      "createdAt": "2016-09-09T22:39:22.000+0000",
      "updatedAt": "2016-09-09T22:39:22.000+0000",
      "labelSummary": {
           "labels": [
          {
              "id": 37,
              "datasetId": 24,
              "name": "beach",
              "numExamples": 49
          },
          {
            "id": 36,
            "datasetId": 24,
            "name": "mountain",
            "numExamples": 50
          }
        ]
      },
      "totalExamples": 99,
      "totalLabels": 2,
      "available": true,
      "statusMsg": "SUCCEEDED",
      "type": "image",
      "object": "dataset"
    },
    {
      "id": 25,
      "name": "Brain Scans",
      "createdAt": "2016-08-24T21:35:27.000+0000",
      "updatedAt": "2016-08-24T21:35:27.000+0000",
      "totalExamples": 0,
      "totalLabels": 0,
      "available": true,
      "statusMsg": "SUCCEEDED",
      "type": "image",
      "object": "dataset"
    }
  ]
}

Generate a token in Ruby

To generate a token in your Ruby application, you include the jwt and rest-client gem in your Gemfile.

gem 'jwt'
gem 'rest-client', '~> 1.8'

Run bundle install to download and resolve all dependencies.

Generate the Token

require 'jwt'
require 'rest-client'
require 'json'

ps_endpoint = ENV['EINSTEIN_VISION_URL']
subject = ENV['EINSTEIN_VISION_ACCOUNT_ID']
private_key = String.new(ENV['EINSTEIN_VISION_PRIVATE_KEY'])
private_key.gsub!('\n', "\n")
expiry = Time.now.to_i + (60 * 15)

# Read the private key string as Ruby RSA private key
rsa_private = OpenSSL::PKey::RSA.new(private_key)

# Build the JWT payload
payload = {
        :sub => subject,
        :aud => "https://api.einstein.ai/v2/oauth2/token",
        :exp => expiry
    }

# Sign the JWT payload
assertion = JWT.encode payload, rsa_private, 'RS256'
puts assertion

# Call the OAuth endpoint to generate a token
response = RestClient.post(ps_endpoint + 'v2/oauth2/token', {
        grant_type: 'urn:ietf:params:oauth:grant-type:jwt-bearer',
        assertion: assertion
    })

token_json = JSON.parse(response)
puts "\nGenerated access token:\n"
puts JSON.pretty_generate(token_json)

access_token = token_json["access_token"]

See a sample Ruby app here.

Generate a token in Java

To generate a token in your Java application, you add the following dependencies in pom.xml.

<dependency>
  <groupId>org.bitbucket.b_c</groupId>
  <artifactId>jose4j</artifactId>
  <version>0.5.4</version>
</dependency>
<dependency>
  <groupId>commons-codec</groupId>
  <artifactId>commons-codec</artifactId>
  <version>1.9</version>
</dependency>
<dependency>
  <groupId>org.glassfish.jersey.core</groupId>
  <artifactId>jersey-client</artifactId>
  <version>2.25</version>
</dependency>
<dependency>
  <groupId>com.fasterxml.jackson.core</groupId>
  <artifactId>jackson-databind</artifactId>
  <version>2.8.5</version>
</dependency>

Run mvn install to download and resolve all dependencies.

Generate the token

Read the EINSTEIN_VISION_PRIVATE_KEY config var value into the java PrivateKey object.

/**
 * Extracts private key from EINSTEIN_VISION_PRIVATE_KEY contents
 */
private static PrivateKey getPrivateKey() {
  String privateKeyBase64 = System.getenv("EINSTEIN_VISION_PRIVATE_KEY");
  String privKeyPEM = privateKeyBase64.replace("-----BEGIN RSA PRIVATE KEY-----\n", "");
  privKeyPEM = privKeyPEM.replace("\n-----END RSA PRIVATE KEY-----", "");

  // Base64 decode the data
  byte[] encoded = Base64.decodeBase64(privKeyPEM);

  try {
    DerInputStream derReader = new DerInputStream(encoded);
    DerValue[] seq = derReader.getSequence(0);

    if (seq.length < 9) {
      throw new GeneralSecurityException("Could not read private key");
    }

    // skip version seq[0];
    BigInteger modulus = seq[1].getBigInteger();
    BigInteger publicExp = seq[2].getBigInteger();
    BigInteger privateExp = seq[3].getBigInteger();
    BigInteger primeP = seq[4].getBigInteger();
    BigInteger primeQ = seq[5].getBigInteger();
    BigInteger expP = seq[6].getBigInteger();
    BigInteger expQ = seq[7].getBigInteger();
    BigInteger crtCoeff = seq[8].getBigInteger();

    RSAPrivateCrtKeySpec keySpec = new RSAPrivateCrtKeySpec(modulus, publicExp,
    privateExp, primeP, primeQ, expP, expQ, crtCoeff);

    KeyFactory factory = KeyFactory.getInstance("RSA");
    return factory.generatePrivate(keySpec);
  } catch (IOException | GeneralSecurityException e) {
    // Handle failures
  }
  return null;
}

Generate the JWT assertion using the PrivateKey.

/**
 * Generate JWT assertion
 */
public static String generateJWTAssertion(float expiryInSeconds) {
  String email = System.getenv("EINSTEIN_VISION_ACCOUNT_ID");
  PrivateKey privateKey = getPrivateKey();
  final JwtClaims claims = new JwtClaims();
  claims.setSubject(email);
  claims.setAudience("https://api.einstein.ai/v2/oauth2/token");
  claims.setExpirationTimeMinutesInTheFuture(expiryInSeconds / 60);
  claims.setIssuedAtToNow();

  // Generate the payload
  final JsonWebSignature jws = new JsonWebSignature();
  jws.setAlgorithmHeaderValue(AlgorithmIdentifiers.RSA_USING_SHA256);
  jws.setPayload(claims.toJson());
  jws.setKeyIdHeaderValue(UUID.randomUUID().toString());

  // Sign using the private key
  jws.setKey(privateKey);
  try {
    return jws.getCompactSerialization();
  } catch (JoseException e) {
    return null;
  }
}

Call the OAuth endpoint to generate the token.

/**
 * Obtain an access token
 */
public static void main(String args[]) {
  String assertion = generateJWTAssertion(60 * 15);
  Form form = new Form();
  form.param("assertion", assertion);
  form.param("grant_type", "urn:ietf:params:oauth:grant-type:jwt-bearer");

  Entity<Form> entity = Entity.form(form);

  String endpoint = System.getenv("EINSTEIN_VISION_URL");
  Response response = client.target(endpoint + "v2/oauth2/token")
      .request()
      .post(entity);

  try {
    ObjectMapper mapper = new ObjectMapper();
    JsonNode responseJson = mapper.readValue(response.readEntity(InputStream.class), JsonNode.class);
    System.out.println(responseJson);
  } finally {
    response.close();
  }
}

See a sample Java app here.

Generate a token in Node.js

To generate a token in your Node.js application, you include the jsonwebtoken in your package.json.

"dependencies": {
  "jsonwebtoken": "^7.1.9",
  "request": "^2.78.0"
}

Generate the token

var jwt = require('jsonwebtoken');
var request = require('request');

var url = process.env.EINSTEIN_VISION_URL
var private_key = process.env.EINSTEIN_VISION_PRIVATE_KEY
var account_id = process.env.EINSTEIN_VISION_ACCOUNT_ID

var reqUrl = `${url}v2/oauth2/token`;

// JWT payload
var rsa_payload = {
  "sub":account_id,
  "aud":reqUrl
}

var rsa_options = {
  header:{
    "alg":"RS256",
    "typ":"JWT"
   },
   expiresIn: '1h'
}

// Sign the JWT payload
var assertion = jwt.sign(
  rsa_payload,
  private_key,
  rsa_options
);

var options = {
  url: reqUrl,
  headers: {
    'Content-Type': 'application/x-www-form-urlencoded',
    'accept': 'application/json'
  },
  body:`grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer&assertion=${encodeURIComponent(assertion)}`
}

// Make the OAuth call to generate a token
request.post(options, function(error, response, body) {
  var data = JSON.parse(body);
  console.log(data["access_token"]);
});

Sample code

See these code samples to help you get up and running with the service.

  • Node.js
    • Web app that you can deploy to Heroku
    • Upload an image and get a prediction back
    • Uses the GeneralImageClassifier model
  • Java
  • Python
  • Ruby

Share a custom model

Share an add-on containing custom-trained models between multiple apps by attaching the add-on to each app:

# First, fetch the `einstein-vision` add-on identifier from the original app.
$ heroku addons --app $APP_NAME

# Then, attach that add-on to another app.
$ heroku addons:attach $ADD_ON_IDENTIFIER --app $OTHER_APP_NAME

# Finally, configure the custom model ID on the other app.
# (This may take a different form for a different app; we recommend using the `CUSTOM_MODEL_ID` env var.)
$ heroku config:set CUSTOM_MODEL_ID=$MODEL_ID --app $OTHER_APP_NAME

When an Einstein Vision and Language add-on is created, it gets a new Einstein Platform Services account. As custom models are created, they are scoped to that account. To share those models, you can attach the add-on to multiple apps.

Migrating between plans

Use the heroku addons:upgrade command to migrate to a new plan.

Be aware of rate limits associated when you change your plan.

$ heroku addons:upgrade einstein-vision:basic
Changing einstein-vision-curved-87707 on coral-brook-39111 from ... done

Removing the add-on

The Einstein Vision and Language add-on can be removed via the CLI.

This destroys all associated datasets and models and can’t be undone!

$ heroku addons:destroy einstein-vision
Destroying einstein-vision-transparent-56474 on ⬢ coral-brook-39111... done

Support

If you run into any issues using the Einstein Vision and Language add-on or need help, please use the Heroku support channels.

Additional resources

Einstein Platform Services API documentation is available here.

Salesforce AI Research

Einstein.ai API Terms of Service

Your use of the Einstein.ai Service(s) is subject to the following terms of service: http://www.sfdcstatic.com/assets/pdf/misc/salesforce_MSA.pdf (“MSA”), as well as to the following Supplemental Terms, which are hereby incorporated to the MSA by reference. Capitalized terms not defined herein shall have the meaning ascribed to them in the MSA.

Supplemental Terms: “Einstein.ai API” means the API endpoint accessible at https://api.einstein.ai. “Einstein.ai Service(s)” means the Einstein.ai API for the Einstein Vision and Language Service, or any other associated online service accessed by You via the Einstein.ai API. “Prediction” means an API call to the Services with the purpose of generating a prediction. “Beta Services” means SFDC services or functionality that may be made available to Customer to try at its option at no additional charge which is clearly designated as beta, pilot, limited release, developer preview, non-production, evaluation, or by a similar description.

You acknowledge that SFDC may access Your Data submitted to the Einstein.ai Service(s) for the purpose of developing and improving the Einstein.ai Service(s), and any other of SFDC’s current and future similar or related features, products and/or services.

You shall comply with all applicable laws and regulations in Your collection, use, processing and retention of any images, text, sounds and/or other data submitted to the Einstein.ai Service(s), and hereby represent and warrant that You have obtained all consents and provided all notices (including, if applicable, through clear disclosure in privacy policies) that are necessary to use, process, and retain all images, text, sounds and/or other data You submit to the Einstein.ai Service(s).

Sensitive personal data may not be submitted to the Einstein.ai Service(s), including images, text, sounds or other data containing or revealing government-issued identification numbers; financial information (such as credit or debit card numbers, any related security codes or passwords, and bank account numbers); racial or ethnic origin, political opinions, religious or philosophical beliefs, trade-union membership, information concerning health or sex life; information related to an individual’s physical or mental health; and information related to the provision or payment of health care. Additionally, You may not use the Einstein.ai Service(s) to create or analyze biometric identifiers such as face prints or fingerprints, or scans of eyes, hands or facial geometry, nor may You use the Einstein.ai Service(s) for the purposes of analyzing, profiling or targeting someone’s racial or ethnic origin, political opinions, religious or philosophical beliefs, trade union membership, age, gender, sex life, sexual orientation, criminal convictions, disability, health status or medical condition.

In addition: For paying users of Einstein Vision and Language only, the following terms also apply: Your use of the Einstein.ai Service(s) includes 2,000 Predictions per calendar month. You understand that the above limitation is contractual in nature (i.e., it is not limited as a technical matter in the Service(s)) and therefore You agree to strictly review Your use of the Service(s) to conform with the limit set forth above. SFDC may review Your use of the Service(s) at any time through the Service(s). Unused Predictions are forfeited at the end of each calendar month and do not roll over to subsequent calendar months. The beginning and end of each calendar month will conform with UTC Time.

For free users of Einstein Vision and Language only, the following terms also apply: Your use of the Einstein.ai Service(s) includes 2,000 free Predictions per calendar month. SFDC reserves the right to limit the size of databases, the number of Models that You can train and Your general access to the Einstein.ai API. Unused Predictions are forfeited at the end of each calendar month and do not roll over to subsequent calendar months. The beginning and end of each calendar month will conform with UTC Time.

We may update these Einstein.ai API Terms of Service at any time. We’ll post notice of modifications to these terms on this page. We may also notify you of material revisions, for example via a service notification or an email to the email associated with your Salesforce or Heroku account. Your continued use of the Einstein.ai Service(s) means you agree to the updated terms, and if you don’t agree to the updated terms, you must stop using the Einstein.ai API.

Effective June 28, 2017

Feedback

Log in to submit feedback.