
This add-on is operated by Expedited Security
Pull website status and domain information for any URL.
URL X-Ray
Last updated July 28, 2023
This article is a work in progress, or documents a feature that is not yet released to all users. This article is unlisted. Only those with the link can access it.
The URL X-Ray add-on is currently in beta.
Table of Contents
URL X-Ray is an add-on to get the website status, domain information and performance results for any site.
Discover a website’s configuration information, response times, redirects, validity and current status.
This let you:
- Filter sites by technology used
- Check the up-time status of a site over time
- Monitor for site issues
- Check load times for slowness affecting users
- Run HTTP/HTTPS checks
- Determine overall site size
- Optionally return page code for further analysis
URL X-Ray exposes an API and has supported examples for Ruby, PHP, Python, Node and Java.
Provisioning the add-on
URL X-Ray can be attached to a Heroku application via the CLI:
A list of all plans available can be found here.
$ heroku addons:create urlxray
-----> Adding urlxray to sharp-mountain-4005... done, v18 (free)
Once URL X-Ray has been added a URLXRAY_API_KEY
config variable will contain your specific API key granting access to the newly provisioned URL X-Ray instance. This can be confirmed using the heroku config:get
command.
$ heroku config:get URLXRAY_API_KEY
Bl4XHVbdsf5GIXQbqTfXR5IrpuuXER2kaVc2zNnA
After installing URL X-Ray your application should be modified to fully integrate with the add-on.
Local setup
Environment setup
After provisioning the add-on it’s necessary to locally replicate the config vars so your development environment can operate against the service.
Use the Heroku Local command-line tool to configure, run and manage process types specified in your app’s Procfile. Heroku Local reads configuration variables from a .env
file. To view all of your app’s config vars, type heroku config
. Use the following command for each value that you want to add to your .env
file.
$ heroku config:get URLXRAY_API_KEY -s >> .env
Credentials and other sensitive configuration values should not be committed to source-control. In Git exclude the .env
file with: echo .env >> .gitignore
.
For more information, see the Heroku Local article.
Input Parameter Descriptions
All languages will follow this call structure for inputs.
Index | Name | Example | Description |
---|---|---|---|
0 | url |
string | The URL to process |
1 | fetch_content |
boolean | If this URL responds with html, text, json or xml then return the response. This option is useful if you want to perform further processing on the URL content. |
Using with Ruby
Install the Url-X-Ray gem.
# In your Gemfile
gem 'urlx_ray', git: 'https://github.com/mbuckbee/Url-X-Ray-Gem.git'
Making a Request
require 'urlx_ray'
# Note: the 'Controller' here is not a reference to Rails controllers
# but an internal structure, won't interfere with your Rails app and will
# work fine in a standalone ruby app or another framework
> uxr = UrlxRay::APIController.new
> result = uxr.lookup("https://www.expeditedssl.com",false)
Using Results
> result.server_name
=> "nginx/1.4.1"
$ > result.http_ok
=> true
> result.server_country_code
=> "US"
> result.server_city
=> "Ashburn"
> result.query
=> {}
> result.content_encoding
=> ""
> result.server_country
=> "United States"
> result.real
=> true
> result.http_redirect
=> false
> result.url
=> "https://www.expeditedssl.com"
> result.server_hostname
=> "ec2-50-16-223-0.compute-1.amazonaws.com"
> result.valid
=> true
> result.server_region
=> "Virginia"
> result.content_size
=> 23848
> result.http_status_message
=> "OK"
> result.content_type
=> "text/html; charset=utf-8"
> result.load_time
=> 0.082296341
> result.server_ip
=> "50.16.223.0"
> result.http_status
=> 200
> result.url_protocol
=> "https"
> result.url_port
=> 443
> result.url_path
=> ""
Using with PHP
<?php
$ch = curl_init('https://urlxray.expeditedaddons.com/?api_key=' . getenv('URLXRAY_API_KEY') . '&fetch_content=false&url=http%3A%2F%2Fwww.wikipedia.org');
$response = curl_exec($ch);
curl_close($ch);
var_dump($response);
Using with Python
import os
from urllib2 import Request, urlopen
request = Request('https://urlxray.expeditedaddons.com/?api_key=' + os.environ['URLXRAY_API_KEY'] + '&fetch_content=false&url=http%3A%2F%2Fwww.wikipedia.org')
response_body = urlopen(request).read()
print response_body
Using with Node
var request = require('request');
request('https://urlxray.expeditedaddons.com/?api_key=' + process.env.URLXRAY_API_KEY + '&fetch_content=false&url=http%3A%2F%2Fwww.wikipedia.org', function (error, response, body) {
console.log('Status:', response.statusCode);
console.log('Headers:', JSON.stringify(response.headers));
console.log('Response:', body);
});
Using with Java
// Maven : Add these dependencies to your pom.xml (java6+)
// <dependency>
// <groupId>org.glassfish.jersey.core</groupId>
// <artifactId>jersey-client</artifactId>
// <version>2.8</version>
// </dependency>
// <dependency>
// <groupId>org.glassfish.jersey.media</groupId>
// <artifactId>jersey-media-json-jackson</artifactId>
// <version>2.8</version>
// </dependency>
import javax.ws.rs.client.Client;
import javax.ws.rs.client.ClientBuilder;
import javax.ws.rs.client.Entity;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.MediaType;
Client client = ClientBuilder.newClient();
Response response = client.target('https://urlxray.expeditedaddons.com/?api_key=' + System.getenv('URLXRAY_API_KEY') + '&fetch_content=false&url=http%3A%2F%2Fwww.wikipedia.org}')
.request(MediaType.TEXT_PLAIN_TYPE)
.get();
System.out.println("status: " + response.getStatus());
System.out.println("headers: " + response.getHeaders());
System.out.println("body:" + response.readEntity(String.class));
Result Field Descriptions
Attribute | Type | Description |
---|---|---|
valid |
boolean | Is this a valid well-formed URL |
real |
boolean | Is this URL actually serving real content |
http_ok |
boolean | True if this URL responded with an HTTP OK (200) status |
http_status |
integer | The HTTP status code this URL responded with |
http_status_message |
integer | The HTTP status message assoicated with the status code |
http_redirect |
boolean | True if this URL responded with a HTTP redirect |
url |
string | The fully qualified URL. This may be different to the URL requested if http-redirect is True |
url_protocol |
string | The URL protocol (usually http or https) |
url_port |
integer | The URL port |
url_path |
string | The URL path |
query |
map | A key:value map of the URL query parameters |
content |
string | The actual content this URL responded with. This is only set if the ‘fetch-content’ parameter was set |
content_size |
integer | The size of the URL content in bytes |
content_type |
string | The content-type the URL points to |
content_encoding |
string | The encoding type the URL uses |
load_time |
float | The time taken to load the URL content (in seconds) |
server_ip |
string | The IP address of the server hosting this URL |
server_name |
string | The name of the server software hosting this URL |
server_country |
string | Server IP geo-location: full country name |
server_country_code |
string | Server IP geo-location: ISO 2-letter country code |
server_city |
string | Server IP geo-location: full city name (if detectable) |
server_region |
string | Server IP geo-location: full region name (if detectable) |
server_hostname |
string | The server hostname (PTR) |
Dashboard
The URL X-Ray dashboard allows you to monitor your API usage limits.
The dashboard can be accessed via the CLI:
$ heroku addons:open urlxray
Opening urlxray for sharp-mountain-4005
or by visiting the Heroku Dashboard and selecting the application in question. Select URL X-Ray from the Add-ons menu.
Troubleshooting
As a sanity check it is sometimes useful to bypass your app stack and check the endpoint, your API Key and parameters directly.
Test with your browser
# Modify the following to use your actual API Key
https://urlxray.expeditedaddons.com/?api_key=REPLACE_WITH_YOUR_URLXRAY_API_KEY&fetch_content=false&url=http%3A%2F%2Fwww.wikipedia.org
A successful call will return your requested data with a HTTP result code of 200
along with your data. We recommend the JSON Formatter extension as a useful tool.
Your API key can be found on your URL X-Ray dashboard.
Migrating between plans
No downtime or disruption of service will occur as you modify your plans.
Use the heroku addons:upgrade
command to migrate to a new plan.
$ heroku addons:upgrade urlxray:newplan
-----> Upgrading urlxray:newplan to sharp-mountain-4005... done, v18 ($49/mo)
Your plan has been updated to: urlxray:newplan
Removing the add-on
URL X-Ray can be removed via the CLI.
This will destroy all associated data, cannot be undone and will immediately block access to the API
$ heroku addons:destroy urlxray
-----> Removing urlxray from sharp-mountain-4005... done, v20 (free)
Support
All URL X-Ray support and runtime issues should be submitted via one of the Heroku Support channels. Any non-support related issues or product feedback is welcome at support@expeditedaddons.com