Pulling Vine Videos Through Twitter’s API with PHP & jQuery

Most developers have been awaiting the release of Vine's official API. Unfortunately this has yet to be announced. However I've been curious to figure out a workaround to this little problem. Vine.co is a social networking startup much like Instagram, but dedicated to sharing quick videos. People often repost these Vine videos onto Twitter coupled with hashtags.

In this tutorial I want to explain how to build a Vine instant search webapp using jQuery and PHP. We'll be accessing live public tweets and scanning them for Vine.co video links. It's a complicated tutorial but the most difficult tasks always have something worth teaching. You can see my instant search in action from the demo link below.

Download Source Code | Live Demo

Registering the Application

Twitter's new API v1.1 requires every API call to use some form of authentication - even for public tweets. I find this to be ridiculous and counter-intuitive, but you gotta work with what you've got. If you're new to registering applications please take a look at this Stack Overflow post. The search request only needs an API key/secret but it doesn't hurt to include a consumer key/secret as well. It's all free to signup from Twitter's developer center.

Don't be nervous if these words seem like nonsense at first! API development is a long process, and you'll gradually become familiar with such terms through repetition. The other thing we need is a generic Twitter API wrapper. I'm using this library on Github which is 100% free to download.

Once downloaded unzip the directory and locate TwitterAPIExchange.php. This will be needed for another PHP script so we can make direct calls to the Twitter API with much less code. For now my directory structure includes an index.html along with a subfolder named /lib/. Copy the API Exchange script into the lib folder(short for "libraries").

Building the Webpage

Setting up the application in Twitter is free and shouldn't take longer than 10-15 minutes. We'll come back to all the PHP stuff later, but for now I want to go over the page layout. The search form uses an ID #vineapi which is targeted using jQuery. Whenever the form is submitted we stop the initial request and run a custom Ajax request instead.

1
2
3
4
5
6
7
8
9
10
11
12

<div id="w">
  <h1>Vine.co Instant Search</h1>
  <p class="center">Enter a #hashtag or keyword(s) to search for related videos.</p>
  
  <form id="vineapi" name="vineapi" method="post" action="index.html">
    <input type="text" name="hashtag" id="hashtag" tabindex="1" placeholder="ex: #funny">
    <input type="submit" value="Search!" id="submitbtn" tabindex="2">
  </form>
 
  <div id="loader"></div>
  <div id="videos" class="clearfix"></div>
</div><!-- @end #w -->

This is the entirety of my page body found within index.html. Aside from the search form there's also a #loader div which is used during Ajax requests. The #videos container will hold all the internal videos as they get loaded by jQuery. Neither of these divs will show on first pageload, but they are very important to the general interface.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

/** page layout **/
#w {
  display: block;
  margin: 0 auto;
  width: 940px;
  background: #fff;
  padding: 15px;
  -webkit-border-radius: 6px;
  -moz-border-radius: 6px;
  border-radius: 6px;
}
 
#loader {
  display: none;
  text-align: center;
}
 
#videos {
  display: block;
  margin: 0 auto;
  width: 100%;
}
.video {
  display: block;
  float: left;
  width: 280px;
  margin-right: 20px;
  margin-bottom: 25px;
}

These are the most important CSS styles related to content display. The #w container is fixed in the center at 940px. Similarly the #videos container fills a 100% width. But each internal .video container is fitted at 280px floating left. Individual Twitter API requests can return 15 max videos and this equates to five rows with 3 videos each.

I'm also using some general resets based off Eric Meyer's codes. The stylesheet isn't very interesting but you can peruse the file at your own leisure.

Creating PHP Scripts

The jQuery/Ajax request will access two separate PHP scripts in two distinct Ajax calls. The first is named api.php which makes the initial connection into Twitter. Let's delve into this code to see how it works.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

require_once('TwitterAPIExchange.php');
 
$settings = array(
  'oauth_access_token' => "ACCESS_TOKEN_HERE",
  'oauth_access_token_secret' => "ACCESS_TOKEN_SECRET_HERE",
  'consumer_key' => "API_KEY_HERE",
  'consumer_secret' => "API_SECRET_HERE"
);
 
$hashtag = $_GET['hashtag'];
$url = 'https://api.twitter.com/1.1/search/tweets.json';
$getfield = '?q='.urlencode($hashtag).'%20vine';
$requestMethod = 'GET';
 
$twitter = new TwitterAPIExchange($settings);
 
echo $twitter->setGetfield($getfield)->buildOauth($url, $requestMethod)->performRequest();

This first block will connect into the Twitter API using your application credentials. Next we need to keep in mind this PHP script will never be manually viewed, only pulled through jQuery. This means we can pass keyword(s) into the script using a $_GET variable. I'll be taking the input field value and parsing with urlencode() to convert symbols for a nicer query. Also each search automatically includes the keyword "vine" so the results are more geared towards what we need.

You don't need to know about the TwitterAPIExchange object but it is documented on the Github page if you're curious to learn. Just understand this api.php file returns JSON results from a Twitter search directly into JavaScript. Then we parse through the results pulling out any tweets which include a Vine URL.

Scraping Video Thumbnails in PHP

Now the secondary PHP script is a lot more convoluted, especially because Vine has no official API methods. In order to pull video thumbnails we need to scrape the entire HTML page of each video and locate the Vine thumbnail URL. This is done using cURL which is a collection of PHP functions to scrape external content(HTML, XML, etc).

The filename is curlinfo.php which also won't be accessed directly, but only through an Ajax request. From the previous script we have access to tweets with links to Vine video pages. Looping through each video we can pass the link into curlinfo.php and obtain the related video thumbnail.

Please note this script uses more of a hack to obtain the thumbnails, so it will not work all of the time.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

$vineurl = $_POST['vidurl'];
 
function get_data($url) {
 $ch = curl_init();
 $timeout = 5;
 curl_setopt($ch, CURLOPT_URL, $url);
 curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
 curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
 $data = curl_exec($ch);
 curl_close($ch);
 return $data;
}
 
$dataobj = get_data($vineurl);
 
// parse HTML content
$doc = new DOMDocument();
@$doc->loadHTML($dataobj);

Instead of $_GET this script uses $_POST data which is very similar. Parsing the Vine URL into a custom function named get_data() will return full HTML contents of each page. Thankfully PHP has a class named DOMDocument for accessing individual elements in the DOM. So I can pass the full HTML contents into a new object and search for each meta tag.

You can see what we're looking for by going to any Vine.co video page and viewing the HTML source. Each page has a meta tag with the property twitter:image containing a thumbnail shot. This is exactly what we need to pull from the DOM and pass back into jQuery.

1
2
3
4
5
6
7
8
9

$list = $doc->getElementsByTagName('meta');
 
foreach( $list as $item ) {
  $prop = $item->getAttribute('property');
 
  if($prop == 'twitter:image') {
    die($item->getAttribute('content'));
  }
}

The $list variable contains a large collection of meta tag DOM objects. I'm checking if the property attribute matches twitter:image, and if so then we kill the script and output the thumbnail URL directly to the page. Content output through PHP is returned to jQuery's Ajax method so we can work with these URLs in JavaScript.

Handling Ajax Requests

If you're still lost from the PHP code try not to be discouraged. As I mentioned this is definitely a hacker's solution to obtain Vine contents, and PHP can be confusing if you're unfamiliar with the language. All you need to understand is that api.php will return a JSON list of tweets. Using those tweets we can pull out Vine URLs to pass into curlinfo.php and then obtain each video's related thumbnail.

But where does all this stuff happen? In the JavaScript! Actually I'm using the more advanced jQuery library to save time on Ajax requests. I'll break down each request so we can better understand how JS and PHP talk to each other.

1
2
3
4
5
6
7
8
9
10
11

$(function(){
  var $loader = $('#loader');
  var $videos = $('#videos');
 
  $('#vineapi').on('submit', function(e){
    e.preventDefault();
 
    var keywords = $('#hashtag').val();
 
    $loader.show();
    $videos.html('');

These first two variables are used many times throughout the script for targeting the loader div and the videos container. The primary event handler will trigger whenever a user submits the #vineapi form, either by clicking the submit button or hitting 'enter' on the keyboard. e.preventDefault() will stop the form submission and instead allows the use of custom JS code.

The variable keywords will hold the current value of the input field. Whatever the user has typed into the search form will be stored in this variable. Afterwards the loading gif is displayed and we clear out any HTML still within the #videos container from a prior search. Then we dive into Ajax requests.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

$.ajax({
    type: 'GET',
    url: 'lib/api.php',
    data: {'hashtag': keywords},
    dataType: 'json',
    success: function(data){
      // on success loop through tweet data
 
 
      // output message if 0 tweets were found
      if(data.statuses.length === 0) {
        $loader.hide();
        $videos.html('<p>No results found.</p>');
      }

The first Ajax method will connect into api.php with the search keyword(s). This also sets the GET datatype so PHP knows how to receive the variable. Whenever an Ajax request is successful we can run a new function with the return data. This can be named anything, but in my example it's passed into the function as data and stores the entire JSON response.

It's important to first check if there's any tweets at all. If the result is empty then we're hiding the loader and displaying a message to the user. Otherwise we've successfully accessed the Twitter API and pulled a number of Tweets containing Vine videos. Now we have to loop through each one and pull the related video thumbnail.

Looping through Search Results

This large block is the final section of jQuery needed to display results. I think the JS code is a lot easier to understand compared with the PHP files - but I hope you can see how they're all related and build onto each other.

1
2
3
4
5

$.each(data.statuses, function(i,itm) {
    // if the tweet has more than 1 link we do some parsing
    if(itm.entities.urls.length > 0) { 
      var shorturl = itm.entities.urls[0].display_url.substring(0,7);
      var videourl = itm.entities.urls[0].expanded_url;

The data variable contains a JSON object of Twitter statuses which we access through dot notation(data.statuses). The $.each() method is very similar to PHP's foreach() loop where we can pass through multiple Tweets individually. To clarify, itm represents each individual data.statuses[i] where i increments with each new tweet. So to access the URLs of a single tweet we would need something like data.statuses[0].entities.urls.

itm.entities.urls[0] will return the very first URL in the list. If the tweet doesn't have any URLs then we completely skip over that tweet and move onto the next one. I'm checking if the URL does indeed point towards vine.co and if so then we need to check for a video thumbnail. Cue the secondary Ajax request to curlinfo.php.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

if(shorturl == 'vine.co') {
              $.ajax({
                type: 'POST',
                url: 'lib/curlinfo.php',
                data: {"vidurl": videourl},
                dataType: 'html',
                success: function(img){
                  if($loader.is(":visible")) { $loader.hide(); }
 
                  if(img == "") { img = "img/default.jpg" }
 
                  var newhtml = '<div class="video"><a target="_blank" href="'+videourl+'"><img src="'+img+'"></a></div>';
 
                  $videos.append(newhtml);
                }
              });
            } // end logic for video URL        
          }
        }); // end each loop
      }
    }); // end primary ajax call
  }); // end click event handler
});

This time I'll pass the data through POST because URL symbols are tricky with colons and forward slashes. In this case I'm expect a return datatype of HTML instead of JSON, because we only expect 1 image link upon successful Ajax request. If successful the thumbnail URL gets passed via function(img).

I mentioned earlier that scraping the DOM will not always yield perfect results. In the case we get an empty image string I've created a default thumbnail img/default.jpg. Either way this success function creates the final HTML output for the individual video. It's contained inside a .video div with the thumbnail and a link out to the original Vine URL. jQuery's .append() method is perfect for adding each new video into the #videos container before looping over the next one.

This loop of scraping the video page, pulling out a thumbnail and embedding into HTML will continue until we loop through every JSON object.

Download Source Code | Live Demo

Closing

Although I know this may not be useful everywhere, it should help developers understand exactly how useful APIs can be. Once Vine releases their own API this detailed process may not be necessary. But I certainly won't be holding my breath for that day!

You're free to download my source code and rehost this on your own website or customize it to your liking. Just remember the live demo uses my own key/secret, so when downloading a copy you'll need to register your own key/secret for it to work properly. Also since this script requires PHP you'll need to run it on a local or remote web server.