Simultaneuos HTTP requests in PHP with cURL
The basic idea of a Web 2.0-style "mashup" is that you consume data from several services, often from different providers and combine them in interesting ways. This means you often need to do more than one HTTP request to a service or services. In PHP if you use something like file_get_contents() this means all the requests will be synchronous: a new one is fired only after the previous has completed. If you need to make three HTTP requests and each call takes a second, your app is delayed at least three seconds.
Solution
An improvement of course is to cache responses as much as possible, but at one point or another you still need to make those requests.
Using the curl_multi* family of cURL functions you can make those requests simultaneously. This way your app is as slow as the slowest request, as opposed to the sum of all requests. And that's something.
A function
Here's a little function I coded that will allow you do multi requests.
function multiRequest($data, $options = array()) { // array of curl handles $curly = array(); // data to be returned $result = array(); // multi handle $mh = curl_multi_init(); // loop through $data and create curl handles // then add them to the multi-handle foreach ($data as $id => $d) { $curly[$id] = curl_init(); $url = (is_array($d) && !empty($d['url'])) ? $d['url'] : $d; curl_setopt($curly[$id], CURLOPT_URL, $url); curl_setopt($curly[$id], CURLOPT_HEADER, 0); curl_setopt($curly[$id], CURLOPT_RETURNTRANSFER, 1); // post? if (is_array($d)) { if (!empty($d['post'])) { curl_setopt($curly[$id], CURLOPT_POST, 1); curl_setopt($curly[$id], CURLOPT_POSTFIELDS, $d['post']); } } // extra options? if (!empty($options)) { curl_setopt_array($curly[$id], $options); } curl_multi_add_handle($mh, $curly[$id]); } // execute the handles $running = null; do { curl_multi_exec($mh, $running); } while($running > 0); // get content and remove handles foreach($curly as $id => $c) { $result[$id] = curl_multi_getcontent($c); curl_multi_remove_handle($mh, $c); } // all done curl_multi_close($mh); return $result; }
Consuming
The function accepts an array of URLs to hit and optionally an array of cURL options if you need to pass any. The first array can be a simple indexed array or URLs or it can be an array of arrays where the second has a key named "url". If you use the second way and you also have a key called "post", the function will do a post request.
The function returns an array of responses as strings. The keys in the result array match the keys in the input.
A GET example
Let's say you want to use some Yahoo search web services (consult YDN) to create a music artist band-o-pedia kind of mashup. Here's how you can search audio, video and images at the same time:
$data = array( 'http://search.yahooapis.com/VideoSearchService/V1/videoSearch?appid=YahooDemo&query=Pearl+Jam&output=json', 'http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Pearl+Jam&output=json', 'http://search.yahooapis.com/AudioSearchService/V1/artistSearch?appid=YahooDemo&artist=Pearl+Jam&output=json' ); $r = multiRequest($data); echo '<pre>'; print_r($r);
This will print something like:
Array
(
[0] => {"ResultSet":{"totalResultsAvailable":"633","totalResultsReturned":...
[1] => {"ResultSet":{"totalResultsAvailable":"105342","totalResultsReturned":...
[2] => {"ResultSet":{"totalResultsAvailable":10,"totalResultsReturned":...
)
A POST example
There's an interesting Yahoo search service called term extraction which analyses content. It accepts POST requests. Here's how to consume this service with the function above, making two simultaneous requests:
$data = array(array(),array()); $data[0]['url'] = 'http://search.yahooapis.com/ContentAnalysisService/V1/termExtraction'; $data[0]['post'] = array(); $data[0]['post']['appid'] = 'YahooDemo'; $data[0]['post']['output'] = 'php'; $data[0]['post']['context'] = 'Now I lay me down to sleep, I pray the Lord my soul to keep; And if I die before I wake, I pray the Lord my soul to take.'; $data[1]['url'] = 'http://search.yahooapis.com/ContentAnalysisService/V1/termExtraction'; $data[1]['post'] = array(); $data[1]['post']['appid'] = 'YahooDemo'; $data[1]['post']['output'] = 'php'; $data[1]['post']['context'] = 'Now I lay me down to sleep, I pray the funk will make me freak; If I should die before I waked, Allow me Lord to rock out naked.'; $r = multiRequest($data); print_r($r);
And the result:
Array
(
[0] => a:1:{s:9:"ResultSet";a:1:{s:6:"Result";s:5:"sleep";}}
[1] => a:1:{s:9:"ResultSet";a:1:{s:6:"Result";a:3:{i:0;s:5:"freak";i:1;s:5:"sleep";i:2;s:4:"funk";}}}
)
This entry was posted on Tuesday, February 19th, 2008 and is filed under mashup, performance, php, yahoo, ydn. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.
Get notification for future posts: follow me on Twitter or subscribe to my RSS feed
February 19th, 2008 at 2:27 am
[...] wrote an interesting post today onHere’s a quick excerptIn PHP if you use something like file_get_contents() this means all the requests will be synchronous: a new one is fired only after the previous has completed. If you need to make three HTTP requests and each call takes a second, … [...]
February 19th, 2008 at 2:53 am
I just use Yahoo! Pipes to pull all my feeds in parallel and give me a single json feed of the processed data. Remind me to write a blog post about it sometime.
Also, ask me tomorrow about another way to do federated queries.
February 19th, 2008 at 8:41 am
Stoyan Stefanov’s Blog: Simultaneous HTTP requests in PHP with cURL…
On his blog today, Stoyan Stefanov has a howto posted on ……
February 19th, 2008 at 11:09 am
[...] On his blog today, Stoyan Stefanov has a howto posted on a trick he figured out to get a PHP script to grab data from multiple resources at one time - with cURL. The basic idea of a Web 2.0-style “mashup” is that you consume data from several services, often from different providers and combine them in interesting ways. This means you often need to do more than one HTTP request to a service or services. […] Using the curl_multi* family of cURL functions you can make those requests simultaneously. This way your app is as slow as the slowest request, as opposed to the sum of all requests. And that’s something. [...]
February 19th, 2008 at 2:36 pm
[...] Stoyan Stefanov wrote a nice article with example code showing just how to accomplish thi. I cannot wait to give it a shot and boost the performance of my apps. Thanks for pointing this out Stoyan. Websites, Useful, Tips & Tricks, News, Raves, Entertainment, PHP. [...]
February 19th, 2008 at 5:02 pm
It would be great trio with SimpleXML and Google Geo API
February 20th, 2008 at 3:46 am
AFAIK it’s possible to make async requests without libcurl: http://php.net/stream_socket_client
February 20th, 2008 at 7:20 am
This article is like gold dust to me. I’m working on an app that is severly limited by curl’s synchronous requests. I’ve had to implement forking to improve speed/scalability. I will be trying this method soon.
Any idea if there’s something similiar for soap requests? If I can crack that i’ll be laughing.
February 22nd, 2008 at 2:34 am
@author
Thanks for bringing attention to this very powerful set of functions- it news to me!
@James
SOAP is merely an http request, so of course you can do it with the cURL-multi. Now, can you use PHP 5’s built-in SOAP extensions? I doubt it. But just use an existing SOAP lib (there’s several good ones) or create your own, it really isn’t that hard. I had to make one for PHP4 awhile back and it was fun.
February 22nd, 2008 at 3:45 pm
[...] simultaneous http requests in php with curl [...]
February 23rd, 2008 at 1:03 pm
Nice article, I going to translate it to russian, if you don’t mind. I think you have a misprint: Simultaneuos -> Simultaneous.
And offtopic: I’ve start a blog-game, and I want to pass on the baton to you. The thing is you need to write about five things/tools, that you can’t work with out it properly. For example it maybe favourite pen, Firefox or MacOS. And next you need to push the baton to 5 blogs from your blogroll.
February 25th, 2008 at 2:21 am
[...] http://www.phpied.com/simultaneuos-http-requests-in-php-with-curl/ [...]
February 25th, 2008 at 4:19 am
Hey Stoyan! Very nice writeup. I didn’t know about these capabilities of curl! Going to try this soon. Thanks
February 28th, 2008 at 3:03 am
[...] [...]
March 11th, 2008 at 1:33 pm
This is EXACTLY what I needed. Thanks! Been pulling my hair out trying to increase the performance of my app. Is there anyway to tie an ID to each request? So that I can track the requests based upon an arbitrary ID as opposed to just the array key?
March 14th, 2008 at 1:59 pm
At last someone did it! We used python for these tasks, but now I can use my lovely PHP for this! Yes!!! Thanks!
March 17th, 2008 at 7:48 pm
Thanks very much for this - I was having trouble getting my head around this, much clearer now. The project I’m sketching up now will definitely benefit from making requests simultaneously…
Cheers, Andi
March 18th, 2008 at 3:17 pm
[...] unknown wrote an interesting post today onHere’s a quick excerpt… that you consume data from several services, often from different providers and combine them in interesting ways. This means you often need to do more than one HTTP request to a service or services. In PHP if you use something like … [...]
April 21st, 2008 at 4:14 pm
Works perfectly thanks!
May 29th, 2008 at 11:22 pm
[...] I recently completed a project that required moving small amounts of data to different servers. Easy enough. The only complication was that it needed to make around 500 separate requests a minute (and growing), to different servers, with different data. Unfortunately cURL typically makes it requests in a synchronous fashion–the next event doesn’t fire until the previous one has completed. When you’re running 500 a minute, every minute, and you have to wait for the previous event to finish, you immediately start building a stockpile of requests–which is bad. Very bad. Every minute the queue grows bigger, and the performance becomes worse. However, cURL has a built in group of functions called curl_multi which allows you to send all of your requests simulatenously. It reduced the processing time so dramatically, that my software can now easily send the 500 requests in under 10 seconds. To that end, I ended up rewriting a function I found over at phpied.com to handle a variety of different scenarios involving post and get parameters. Most of the documentation you need can be found in the example.php file where I included a bunch of different case scenarios. [...]
May 30th, 2008 at 11:28 am
Great tip, I am going to use it on my spiders on wordsfinder.com … I have got a few intelligence tools that might like your findings.
July 13th, 2008 at 5:43 pm
[...] http://www.phpied.com/simultaneuos-http-requests-in-php-with-curl/ [...]
July 18th, 2008 at 3:24 am
[...] here to go to his post Tags: curl, multiple requests, phpRead More Comments (0)php [...]
July 30th, 2008 at 7:02 am
Great job,
I was having CURL timeout issue while making http requests to more than 10-15 different services and it was taking too long to get the responses form all services. Initially i thought to create child processes and treat them as thread (but php has no support for threading in its core so we have to rely on the system function to create processes) but this article solved my problem as i didn’t know CURL support for simultaneous http requests. But now i can use this approach to solve the timeout and child processes issue.
Thanks and once again nice Job.
October 6th, 2008 at 1:38 pm
Just a note on the asynchronous nature of cURL’s multi functions: the DNS lookups are not (as far as I know today) asynchronous. So if one DNS lookup of your group fails, everything in the list of URLs after that fails also. We actually update our hosts.conf (I think?) file on our server daily in order to get around this. It gets the IP addresses there instead of looking them up. I believe it’s being worked on, but not sure if it’s changed in cURL yet.
October 8th, 2008 at 11:13 am
Thanks you for this example of multi curl …
October 10th, 2008 at 4:45 pm
Could you place your code into a downloadable zip’d file? I like the way your website works with these comments. Nice job.
October 11th, 2008 at 12:04 pm
Is it possible to show the results one by one on the website. Cause now firefox just load the page and I have to wait till all pages are collected to get the results and I want to show ‘please wait…’ command, then when results are collected they are inserted one by one, and when finish ‘thank you!’ label is shown. Is it possible to do such thing with these scripts? Or I have to use AJAX or AHAH technology? Thanks, Mark
October 16th, 2008 at 9:28 pm
[...] posted by iwatani on 10 月 17 « Google&Yahoo&Amazon一括検索 先日、マッシュアップに必須！PHPで複数APIを同時に叩いて超高速化するサンプル という記事を発見しました。 概要としては 通常APIを複数利用するときなどはAPIにリクエストし、結果が返ってきたら次のAPIにリスエストを行いますが multiRequest関数を利用すると複数のAPIに同時にリクエストを行うため高速化されるという事です。 早速、試しに便利ツールTipsにあるRank Checker(Google|Yahoo|MSN|Baidu)を改良して実験してみます。今回は開発環境でsymfonyのレンダリングを使って計測します。 Rank Checker(Google|Yahoo|MSN|Baidu)はそれぞれの検索エンジンで検索結果を取得し、URLの順位を拾って表示しています。上位から順に確認して検索結果でURLが見つかった時点で次の動作に移るようにしてあるので検索結果にURLが見つからない場合が一番処理が長くなってしまいます。 例えば キーワード 三愛企画 URL http://www.iii-planning.com/ をチェックしてみると2281msですが キーワード ネイル URL http://www.iii-planning.com/ だと29806msもかかってしまいます。 This entry was posted on 金曜日, 10 月 17th, 2008 at 11:28 AM and is filed under 未分類. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site. [...]
October 29th, 2008 at 9:11 pm
Thanks for your tip. I would like to translate to Vietnamese if you don’t mind. I made a pingback.
December 3rd, 2008 at 5:37 am
I usually don’t comment but this one deserves a thumb up. You’ve done a great job explaining this curl_multi* family as you name it.
January 18th, 2009 at 11:31 am
[...] that will get some information from e.g. 5 sites simultaneously. I tried the following script: http://www.phpied.com/simultaneuos-h…php-with-curl/ Everything works fine, but what I want is simultaneuos (something to multithread, when these 5 [...]
January 18th, 2009 at 12:58 pm
works perfect thanks…
January 26th, 2009 at 8:35 pm
Thanks for sharing this example. I made some modifications so that you can process each request as soon as it completes. It makes things a lot faster when you’re dealing with a large number of requests:
http://onlineaspect.com/2009/01/26/how-to-use-curl_multi-without-blocking/
February 3rd, 2009 at 6:41 am
Very nice. Thanks for your example. Useful.
February 11th, 2009 at 12:22 am
// execute the handles
$running = null;
do {
curl_multi_exec($mh, $running);
} while($running > 0);
Don’t use it without some little delay!
Better way - insert inside cycle:
usleep(25000);
September 1st, 2009 at 5:03 am
I had a problem before understood that such web requests to APIs must be simultaneous. It sometimes takes up to 3 minutes for my app to get and parse data from remote server. Hopefully this function will help to significantly decrease page load time. Thank you for solution!
September 3rd, 2009 at 1:51 pm
[...] This post was Twitted by phpprofession [...]
October 2nd, 2009 at 11:59 pm
[...] by making it very fast and easy to browse the code documentation along with the code itself. Simultaneuos HTTP requests in PHP with cURL / phpied.com Mehrere HTTP Request parallel mit cURL in PHP? Hier wird beschrieben wie! Share this on [...]
January 23rd, 2010 at 9:15 am
I don’t understand the loop. Why would you want to loop curl_multi_exec()? It only needs to be run once… What am I missing? Thanks.
// execute the handles
$running = null;
do {
curl_multi_exec($mh, $running);
usleep(25000);
} while($running > 0);