MySQL question

Excellent ibis.


Time out Exec script

Fatal error: Maximum execution time of 300 seconds exceeded in /home/silverpo/public_html/timmy/timeoutexec.php on line 25

http://www.silverpond.eu/timmy/timeoutexec.php

That is what I call 'Fucking Cool' . ;

EvotingMachine0197 wrote:That is what I call 'Fucking Cool' . ;

Always glad to be of service!

ibis wrote:

Kate P wrote:ibis wrote

You can also use (as I often do) Windows-IIS-mySQL-PHP, which makes one a WIMPy programmer.

This playing around with computer languages is fun afterall...

Cheeky - I'll have you know you're using my intellectual property there, and modifying it without my permission. I coined WIMP back in 2001, during a training course I was running!

Mea culpa - my mistake, ibis was in assuming after your more than robust defence of the Lisbon treaty, that there could possibly be anything WIMPy about you.

Threads like this, on a more serious note, always make me conscious of how little I know about how the very machine I'm working on now, actually works. My secondary school was one of the first in the country to have a computer lab - of Amstrads /Commodore 64s. I learned Basic in second year and nothing since. But it is interesting to observe you and EVM conversing in a totally different language that to me just illustrates the pace of change in the computer industry in the last 20 years.

TheGeniusOfCork wrote:
I must really be a lamplighter, because I can add Perl, Python, Flash Graphics Design, Actionscript, ObjectC, Squeak SmallTalk, and of course the oldies, Fortran, C, C++, Rexx, Pascal, Basic, oh! and Blender the 3-d Graphic Design system, and of course Mathematica, Linpack,... oh. when will I stop bragging.

I

Eh...never, you are from feckin' Cork after all!

Ah the good old days - reading through my Oracle SQL reference book, debugging the query generation code, talking to my screen - simple pleasures.

EvotingMachine0197 wrote:Well then , coiner of WIMP, is it possible to leave a continuously running PHP script on a host server ? ie. if I make a script that say .. has to load 10000 website homepages in turn and search for a string - and if (string exists) {save url to file}

Yes you could do that but the more elegant approach would be to use the curl function to copy each homepage and then process the data in the background. You will find that of the 10K websites, some will no longer exist, some will have moved (301/302 results) and others may be forbidden/coming soon/pointed to a parking page. Normally the databases here check millions of websites when surveying TLDs such as ie/eu/co.uk etc (in addition to tracking over 210 million active and inactive domains (2000-2008)). You can run php scripts from the command line but you've got to constantly monitor scripts like this to make sure that they don't go wild and the way that some idiots write code, they can unexpectedly crash even the best written crawlers and spiders.

This could take hours conceiveably (spell) ?

At least a few hours at least if you do it linearly. The timeouts and the missing websites will increase the time taken. The mistake that a lot of people seem to make is to try to process the data when the webpage collection program is running.

Is it normal for a host server to put a time limit on execution time ?

It would be best to have a timeout inside the script rather than relying on the global script execution limit. With curl you can set that time out and I think that most of the curl functions are available in PHP.

If you need to read a book about this kind of stuff, the O'Reilly "Spidering Hacks" book is a good one though it is heavily reliant on Perl rather than PHP.

Regards...jmcc

Thanks jmcc. I have not used curl before so I checked it out on php.net. It does seem to have a fairly powerful function set in PHP.

Which leads nicely to something eles I wanted to do, which was to get my server to behave as a client and log into another website ??
This introduces the whole area of HTTP POST and cookies and all that stuff which I haven't got my head around yet.

EvotingMachine0197 wrote:Thanks jmcc. I have not used curl before so I checked it out on php.net. It does seem to have a fairly powerful function set in PHP.

Which leads nicely to something eles I wanted to do, which was to get my server to behave as a client and log into another website ??
This introduces the whole area of HTTP POST and cookies and all that stuff which I haven't got my head around yet.

You really will need curl for that! It's certainly doable though - about ten lines of code.

There should be a few tutorials on using Curl and curl in php to log into sites. But the book I mentioned earlier is one of the best introductions that I've seen (there is not a lot of books on the subject as spider/crawler development tends to be a very specialised thing). Though as Ibis said above, it is only a few lines of code to get it working.

Another important thing is to be wary of how your script handles HTTP codes. Sometimes you can find that web admins create their own HTTP codes (0, 999 or some other weird code) and if your script is expecting a real result code, it may hang - hence the need for a timeout and a HTTP result code parser with a default exit code.

Regards...jmcc