Whitelist is available to everyone. Happy 2015!

iparamonau · iparamonau · commit 189b05f0209a · 2014-12-29T14:53:50.000-05:00
diff --git a/README.md b/README.md
@@ -1,35 +1,48 @@
 # Iframely API for Responsive Web Embeds
 
-Iframely is fast and simple HTTP API for responsive web embeds. It also can be used as Node.js library. 
+Iframely is fast and simple HTTP API for responsive web embeds and semantic meta. The API covers well [over 1600 domains](https://iframely.com/try) through 150 custom domain plugins and generic parsers for [oEmbed](http://oembed.com/), [Open Graph](http://ogp.me/) and [Twitter Cards](https://dev.twitter.com/docs/cards), powered by Iframely's whitelist. 
 
-HTTP API are available as [oEmbed](https://iframely.com/docs/oembed-api) or [Iframely API](https://iframely.com/docs/iframely-api) formats. Iframely formats basically mimics the `<head>` section of the page with its `meta` and `links` elements.
+The whitelisting is manual process, and we do every-day manual regression testing to ensure integrity. [Submit your domain](https://iframely.com/qa/request), if you publish embeds.
+
+HTTP API are available as [oEmbed](https://iframely.com/docs/oembed-api) or [Iframely API](https://iframely.com/docs/iframely-api) formats. To make it simple to understand, Iframely format mimics the `<head>` section of the page with its `meta` and `links` elements.
 
 In response to `url` request, APIs returns you the embeds and meta for a requested web page. Below are samples from [hosted API](https://iframely.com), just to show you the format:
 
 - [>> Here’s API call for Coub video](http://iframe.ly/ACcM3Y.json)
 - [>> Same one, but as oEmbed](http://iframe.ly/ACcM3Y.oembed)
 
-Iframely does it by parsing [oEmbed](http://oembed.com/), [Open Graph](http://ogp.me/) and [Twitter Cards](https://dev.twitter.com/docs/cards) and general meta on the original page. Or by using specific domain plugins in the package. 
+Iframely can also be used as Node.js library. 
+
+
+## Get started:
 
-There are over 150 custom domains included as open-source. Plus, there is a whitelist option for oEmbed, Open Graph and Twitter Cards that acts like a gigantic plugin for generic parsers. You can [create your own whitelist](https://iframely.com/docs/whitelist-format) or [get one](https://iframely.com/plans) with [over 1600 domains](https://iframely.com/try) from Iframely. By default, our whitelist with Top 100 domains is included for free with your package. 
+The simplest way to see Iframely in action is by using our [Chrome plugin](https://chrome.google.com/webstore/detail/iframely-url-previews/bbafbcjnlgfbemjemgliogmfdlkocjmi). This way you will see the embeds on your (or any really) site right away (but not on GitHub due to Content-Security-Policy, sorry). 
 
-## Read Next:
+Or, [try Iframely demo with your very own or any other Twitter feed](https://iframely.com/try). Don't forget to click on results to see the embed codes.
 
- - [Try Iframely demo with any Twitter feed](https://iframely.com/try)
+To get started with the APIs: 
+
+ - There's a hosted version of these APIs at [iframely.com](https://iframely.com), if you'd rather rely on the cloud
  - [API in Iframely format](https://iframely.com/docs/iframely-api) (`iframe=true` option is only available for hosted API)
  - [API in oEmbed format](https://iframely.com/docs/oembed-api)
- - [About Link Rels, Types and Media Queries](https://iframely.com/docs/links) (players, thumbnails, app, reader, survey, slideshow, etc)
+ - [About Link Rels, Types and Media Queries](https://iframely.com/docs/links) in Iframely format (players, thumbnails, app, reader, survey, slideshow, etc)
  - [META semantics](https://iframely.com/docs/meta) Iframely API scrapes for you.
  - [How to install & configure](https://iframely.com/docs/host) your open-source host. 
 
+
+## Use APIs in your open-source project
+
+We have provided specific endpoint for the use in open source projects. Read more at [oembedapi.com](http://oembedapi.com).
+
+
+
 ## Contribute
 
 We put our best effort to maintain Iframely and all its domain parsers. Please, feel free to [reach us on Twitter](http://twitter.com/iframely) or to [submit an issue](https://github.com/itteco/iframely/issues) if you have any suggestions. Our support email is support at iframely.com
 
 Fork and pull-request, if you'd like to add more plugins and/or contribute fixes or improvements. By doing so, you make your work available under the same MIT license.
 
-If you are a publisher and would like to make your embeds available under [Iframely Protocol](http://iframely.com/oembed2) (and thus distributed through this open-source gateway) - please, [add your domain to the our DB](http://iframely.com/qa/request).
-
+If you see an error in our domains whitelist (you can [debug URLs here](http://iframely.com/debug)), please ping us and we'll fix it in no time.
 
 
 ## License & Authors
diff --git a/config.js b/config.js
@@ -7,7 +7,7 @@
 
     var config = {
 
-        WHITELIST_URL: 'http://iframely.com/qa/top100.json',
+        WHITELIST_URL: 'http://iframely.com/qa/whitelist.json',
         WHITELIST_URL_RELOAD_PERIOD: 60 * 60 * 1000,  // will reload WL every hour, if no local files are found in /whitelist folder
 
         WHITELIST_LOG_URL: 'http://iframely.com/whitelist-log',
diff --git a/config.local.js.SAMPLE b/config.local.js.SAMPLE
@@ -14,11 +14,6 @@
 
         port: 8061,
 
-        // You can set your unique whitelist access url here.
-        // Iframely will periodically load WL updates from remote file.
-        // To get one - go to http://iframely.com/qa
-        WHITELIST_URL: 'http://iframely.com/qa/top100.json',
-
         // Optional SSL cert, if you serve under HTTPS. 
         /*
         ssl: {
@@ -115,12 +110,12 @@
             }
         },
 
-        // WHITELIST_WILDCARD, if present, will be added to whitelist (even absent one) as record for top level domain: "*"
-        // you can change default behavior of the system using this setting. 
-        // E.g. allowing all open-graph players, denying twitter photos, etc.
-        // If absent or empty, all generic media parsers will be disabled by default (only domain plugins will work)
-        // More about format: http://iframely.com/qa/format
+        // WHITELIST_WILDCARD, if present, will be added to whitelist as record for top level domain: "*"
+        // with it, you can define what parsers do when they run accross unknown publisher.
+        // If absent or empty, all generic media parsers will be disabled except for known domains
+        // More about format: https://iframely.com/docs/qa-format
 
+        /*
         WHITELIST_WILDCARD: {  
               "twitter": {
                 "player": "allow",
@@ -133,7 +128,7 @@
                 "link": "deny"
               },
               "og": {
-                "video": ["allow", "ssl"]
+                "video": ["allow", "ssl", "responsive"]
               },
               "iframely": {
                 "survey": "allow",
@@ -145,6 +140,7 @@
                 "video": ["allow", "responsive"]
               }
         }
+        */
     };
 
     module.exports = config;
diff --git a/docs/SETUP.md b/docs/SETUP.md
@@ -41,11 +41,7 @@ At the very least, you need to properly configure:
 - `allowedOrigins` - very important to list your main app's domain(s) here, and block access to others 
 
 
-If you got Iframely whitelist with [Domains DB](https://iframely.com/qa), configure the direct link to it as `WHITELIST_URL`. (Default points to Top 100 domains list).
-
-Otherwise, the important piece to configure is `WHITELIST_WILDCARD`. This record indicates the default behavior of the the generic parsers with regards to various embeds protocols and types. For example, you can allow or deny Open Graph videos, any oEmbed types or Twitter Players. If you leave this record empty or omit it alltogether, no additional rich parsers will be enabled, leaving domain providers, meta and thumbnails ones only. See the [record format description](https://iframely.com/qa/format).
-
-There are also some provider-specific values you might want to configure (e.g. wheather to include media in Twitter status embeds). Please, enter your own application keys and secret tokens where applicable. Parsers for Twitter, Flickr and Tumblr won't work without the access tokens to their APIs.
+There are also some provider-specific values that are required for some domain plugins to work (e.g. wheather to include media in Twitter status embeds). Please, enter your own application keys and secret tokens where applicable. Parsers for Twitter, Flickr and Tumblr won't work without the access tokens to their APIs.
 
 
 Readability parser to get the HTML of the articles is optional and is turned off by default as it affects the processing time of the URLs. If need be, you can also fine-tune API response time by disabling image size detection.
@@ -76,9 +72,9 @@ For production deployments, we recommend the cluster mode. It will properly util
 Depending on your setup, you may need to configure these pathes in your reverse proxy settings to point to Iframely's Node.js instance:
 
     /r/.+               -- static files (including iframely.js client library)
-    /iframely           -- main API endpoint with get params - returns oEmbed/2 as JSON
-    /oembed             -- wrapper around main API. Returns oEmbed v1 JSON and other meta
-    /debug              -- optional debugger UI with get params
+    /iframely           -- main API endpoint with get params 
+    /oembed             -- oEmbed API endpoint
+    /debug              -- optional debugger UI, if you write your own domain plugins
     /reader.js          -- API endpoint with get params - proxies script to render article
     /render             -- API endpoint with get params - prexies custom widgets if required
     /meta-mappings      -- optional API endpoint with available unified meta
@@ -105,16 +101,5 @@ and restart your server afterwards. If you use [Forever](https://github.com/node
 If you fork, make sure to merge from the upstream for the newer versions.
 
 
-## Extend functionality with Domains DB
-
-You can greatly extend gateway functionality without writting additional plugins. Just upload Domains DB JSON file into `whitelist` folder and Iframely will start covering extra domains, generating responsive players, twitter photos, etc via generic plugins.
-
-The file name is expected to be of "iframely-*.json" pattern. Lastest filename uploaded to this directory prevails. 
-
-You can get whitelist file with over 1600 domains at [iframely.com/qa/buy](https://iframely.com/qa/buy). Setting `WHITELIST_URL` in config file to your personal access URL will instruct Iframely to load domains DB from the server periodically. If neither local file nor `WHITELIST_URL` are provided, Iframely will use a free file with top 100 domains from [iframely.com/qa/top100.json](http://iframely.com/qa/top100.json). 
-
-If you wish to create your own whitelist, please, follow [required file format](http://iframely.com/qa/format).
-
-
 
 (c) 2013 [Itteco Software Corp](http://itteco.com). Licensed under MIT. [Get it on Github](https://github.com/itteco/iframely)
diff --git a/lib/whitelist.js b/lib/whitelist.js
@@ -273,7 +273,7 @@
         if (filename && filename != currentWhitelistFilename) {
             readWhitelist(filename);
         } else {
-            console.log('No local Domains DB whitelist file detected...');
+            console.log('No local whitelist file detected...');
             addWildcard();
         }
 
@@ -313,15 +313,7 @@
 
         if (!currentWhitelistFilename && CONFIG.WHITELIST_URL && CONFIG.WHITELIST_URL_RELOAD_PERIOD) {
 
-            if (CONFIG.WHITELIST_URL.search(/top100\.json/) > -1) {
-                console.log("");
-                console.log("Loading free whitelist with top 100 domains from " + CONFIG.WHITELIST_URL);
-                console.log("*** Full version (w/ over 1500 domains): http://iframely.com/qa/buy");
-                console.log("*** Or hosted & All-in-One API: https://iframely.com");
-                console.log("");
-            } else {
-                console.log("Loading whitelist from " + CONFIG.WHITELIST_URL);
-            }
+            console.log("Loading whitelist from " + CONFIG.WHITELIST_URL);
 
             request({
                 uri: CONFIG.WHITELIST_URL,
diff --git a/whitelist/README.md b/whitelist/README.md
@@ -1,9 +1,7 @@
-If you put a whitelist json file here, Iframely will start covering extra domains, converting responsive players, photos, etc. 
+If you put your own whitelist json file here, Iframely will use it to covere domains through oEmbed, Open Graph and Twitter Cards.
 
 The file name is expected to be of "iframely-*.json" pattern. Lastest filename uploaded to this directory is used. 
 
-- You can get whitelist file with over 2000 domains at [http://iframely.com/qa/buy](http://iframely.com/qa/buy). 
-- Or get free extract file with top 100 domains from the same page.
-- Configure `WHITELIST_URL` and `WHITELIST_URL_RELOAD_PERIOD` with your unique access link to download files automatically
+Please, follow this [file format](http://iframely.com/qa/format).
 
-If you wish to create your own whitelist, please, follow this [file format](http://iframely.com/qa/format).
+By default, the package uses Iframely's whitelist with over 1600 domains. We do manual QA every day, so it might be wise of you to rely on us for whitelisting.
