Simultaneuos HTTP requests in PHP with cURL
The basic idea of a Web 2.0-style "mashup" is that you consume data from several services, often from different providers and combine them in interesting ways. This means you often need to do more than one HTTP request to a service or services. In PHP if you use something like file_get_contents() this means all the requests will be synchronous: a new one is fired only after the previous has completed. If you need to make three HTTP requests and each call takes a second, your app is delayed at least three seconds.
Solution
An improvement of course is to cache responses as much as possible, but at one point or another you still need to make those requests.
Using the curl_multi* family of cURL functions you can make those requests simultaneously. This way your app is as slow as the slowest request, as opposed to the sum of all requests. And that's something.
A function
Here's a little function I coded that will allow you do multi requests.
function multiRequest($data, $options = array()) { // array of curl handles $curly = array(); // data to be returned $result = array(); // multi handle $mh = curl_multi_init(); // loop through $data and create curl handles // then add them to the multi-handle foreach ($data as $id => $d) { $curly[$id] = curl_init(); $url = (is_array($d) && !empty($d['url'])) ? $d['url'] : $d; curl_setopt($curly[$id], CURLOPT_URL, $url); curl_setopt($curly[$id], CURLOPT_HEADER, 0); curl_setopt($curly[$id], CURLOPT_RETURNTRANSFER, 1); // post? if (is_array($d)) { if (!empty($d['post'])) { curl_setopt($curly[$id], CURLOPT_POST, 1); curl_setopt($curly[$id], CURLOPT_POSTFIELDS, $d['post']); } } // extra options? if (!empty($options)) { curl_setopt_array($curly[$id], $options); } curl_multi_add_handle($mh, $curly[$id]); } // execute the handles $running = null; do { curl_multi_exec($mh, $running); } while($running > 0); // get content and remove handles foreach($curly as $id => $c) { $result[$id] = curl_multi_getcontent($c); curl_multi_remove_handle($mh, $c); } // all done curl_multi_close($mh); return $result; }
Consuming
The function accepts an array of URLs to hit and optionally an array of cURL options if you need to pass any. The first array can be a simple indexed array or URLs or it can be an array of arrays where the second has a key named "url". If you use the second way and you also have a key called "post", the function will do a post request.
The function returns an array of responses as strings. The keys in the result array match the keys in the input.
A GET example
Let's say you want to use some Yahoo search web services (consult YDN) to create a music artist band-o-pedia kind of mashup. Here's how you can search audio, video and images at the same time:
$data = array( 'http://search.yahooapis.com/VideoSearchService/V1/videoSearch?appid=YahooDemo&query=Pearl+Jam&output=json', 'http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Pearl+Jam&output=json', 'http://search.yahooapis.com/AudioSearchService/V1/artistSearch?appid=YahooDemo&artist=Pearl+Jam&output=json' ); $r = multiRequest($data); echo '<pre>'; print_r($r);
This will print something like:
Array
(
[0] => {"ResultSet":{"totalResultsAvailable":"633","totalResultsReturned":...
[1] => {"ResultSet":{"totalResultsAvailable":"105342","totalResultsReturned":...
[2] => {"ResultSet":{"totalResultsAvailable":10,"totalResultsReturned":...
)
A POST example
There's an interesting Yahoo search service called term extraction which analyses content. It accepts POST requests. Here's how to consume this service with the function above, making two simultaneous requests:
$data = array(array(),array()); $data[0]['url'] = 'http://search.yahooapis.com/ContentAnalysisService/V1/termExtraction'; $data[0]['post'] = array(); $data[0]['post']['appid'] = 'YahooDemo'; $data[0]['post']['output'] = 'php'; $data[0]['post']['context'] = 'Now I lay me down to sleep, I pray the Lord my soul to keep; And if I die before I wake, I pray the Lord my soul to take.'; $data[1]['url'] = 'http://search.yahooapis.com/ContentAnalysisService/V1/termExtraction'; $data[1]['post'] = array(); $data[1]['post']['appid'] = 'YahooDemo'; $data[1]['post']['output'] = 'php'; $data[1]['post']['context'] = 'Now I lay me down to sleep, I pray the funk will make me freak; If I should die before I waked, Allow me Lord to rock out naked.'; $r = multiRequest($data); print_r($r);
And the result:
Array
(
[0] => a:1:{s:9:"ResultSet";a:1:{s:6:"Result";s:5:"sleep";}}
[1] => a:1:{s:9:"ResultSet";a:1:{s:6:"Result";a:3:{i:0;s:5:"freak";i:1;s:5:"sleep";i:2;s:4:"funk";}}}
)
February 19th, 2008 at 2:27 am
[…] wrote an interesting post today onHere’s a quick excerptIn PHP if you use something like file_get_contents() this means all the requests will be synchronous: a new one is fired only after the previous has completed. If you need to make three HTTP requests and each call takes a second, … […]
February 19th, 2008 at 2:53 am
I just use Yahoo! Pipes to pull all my feeds in parallel and give me a single json feed of the processed data. Remind me to write a blog post about it sometime.
Also, ask me tomorrow about another way to do federated queries.
February 19th, 2008 at 8:41 am
Stoyan Stefanov's Blog: Simultaneous HTTP requests in PHP with cURL…
On his blog today, Stoyan Stefanov has a howto posted on ……
February 19th, 2008 at 11:09 am
[…] On his blog today, Stoyan Stefanov has a howto posted on a trick he figured out to get a PHP script to grab data from multiple resources at one time - with cURL. The basic idea of a Web 2.0-style “mashup” is that you consume data from several services, often from different providers and combine them in interesting ways. This means you often need to do more than one HTTP request to a service or services. […] Using the curl_multi* family of cURL functions you can make those requests simultaneously. This way your app is as slow as the slowest request, as opposed to the sum of all requests. And that’s something. […]
February 19th, 2008 at 2:36 pm
[…] Stoyan Stefanov wrote a nice article with example code showing just how to accomplish thi. I cannot wait to give it a shot and boost the performance of my apps. Thanks for pointing this out Stoyan. Websites, Useful, Tips & Tricks, News, Raves, Entertainment, PHP. […]
February 19th, 2008 at 5:02 pm
It would be great trio with SimpleXML and Google Geo API
February 20th, 2008 at 3:46 am
AFAIK it's possible to make async requests without libcurl: http://php.net/stream_socket_client
February 20th, 2008 at 7:20 am
This article is like gold dust to me. I'm working on an app that is severly limited by curl's synchronous requests. I've had to implement forking to improve speed/scalability. I will be trying this method soon.
Any idea if there's something similiar for soap requests? If I can crack that i'll be laughing.
February 22nd, 2008 at 2:34 am
@author
Thanks for bringing attention to this very powerful set of functions- it news to me!
@James
SOAP is merely an http request, so of course you can do it with the cURL-multi. Now, can you use PHP 5's built-in SOAP extensions? I doubt it. But just use an existing SOAP lib (there's several good ones) or create your own, it really isn't that hard. I had to make one for PHP4 awhile back and it was fun.
February 22nd, 2008 at 3:45 pm
[…] simultaneous http requests in php with curl […]
February 23rd, 2008 at 1:03 pm
Nice article, I going to translate it to russian, if you don't mind. I think you have a misprint: Simultaneuos -> Simultaneous.
And offtopic: I've start a blog-game, and I want to pass on the baton to you. The thing is you need to write about five things/tools, that you can't work with out it properly. For example it maybe favourite pen, Firefox or MacOS. And next you need to push the baton to 5 blogs from your blogroll.
February 25th, 2008 at 2:21 am
[…] http://www.phpied.com/simultaneuos-http-requests-in-php-with-curl/ […]
February 25th, 2008 at 4:19 am
Hey Stoyan! Very nice writeup. I didn't know about these capabilities of curl! Going to try this soon. Thanks
February 28th, 2008 at 3:03 am
[…] […]
March 11th, 2008 at 1:33 pm
This is EXACTLY what I needed. Thanks! Been pulling my hair out trying to increase the performance of my app. Is there anyway to tie an ID to each request? So that I can track the requests based upon an arbitrary ID as opposed to just the array key?
March 14th, 2008 at 1:59 pm
At last someone did it! We used python for these tasks, but now I can use my lovely PHP for this! Yes!!! Thanks!
March 17th, 2008 at 7:48 pm
Thanks very much for this - I was having trouble getting my head around this, much clearer now. The project I'm sketching up now will definitely benefit from making requests simultaneously…
Cheers, Andi
March 18th, 2008 at 3:17 pm
[…] unknown wrote an interesting post today onHere’s a quick excerpt… that you consume data from several services, often from different providers and combine them in interesting ways. This means you often need to do more than one HTTP request to a service or services. In PHP if you use something like … […]
April 21st, 2008 at 4:14 pm
Works perfectly thanks!