Machine Nation
Would you like to react to this message? Create an account in a few clicks or log in to continue.
Machine Nation

Irish Politics Forum - Politics Technology Economics in Ireland - A Look Under The Nation's Bonnet


Devilish machinations come to naught --Milton
 
PortalPortal  HomeHome  SearchSearch  Latest imagesLatest images  RegisterRegister  Log in  GalleryGallery  MACHINENATION.org  

 

 MySQL question

Go down 
Go to page : Previous  1, 2
AuthorMessage
Ex
Fourth Master: Growth
Ex


Number of posts : 4226
Registration date : 2008-03-11

MySQL question - Page 2 Empty
PostSubject: Re: MySQL question   MySQL question - Page 2 EmptyFri Sep 05, 2008 4:20 am

Excellent ibis.


Time out Exec script

Fatal error: Maximum execution time of 300 seconds exceeded in /home/silverpo/public_html/timmy/timeoutexec.php on line 25

http://www.silverpond.eu/timmy/timeoutexec.php


Last edited by EvotingMachine0197 on Fri Sep 05, 2008 4:23 am; edited 1 time in total (Reason for editing : to add link to time out test script.)
Back to top Go down
Ex
Fourth Master: Growth
Ex


Number of posts : 4226
Registration date : 2008-03-11

MySQL question - Page 2 Empty
PostSubject: Re: MySQL question   MySQL question - Page 2 EmptyFri Sep 05, 2008 4:24 am

That is what I call 'Fucking Cool' . Smile ;
Back to top Go down
Guest
Guest




MySQL question - Page 2 Empty
PostSubject: Re: MySQL question   MySQL question - Page 2 EmptyFri Sep 05, 2008 4:52 am

EvotingMachine0197 wrote:
That is what I call 'Fucking Cool' . Smile ;

Always glad to be of service!
Back to top Go down
Guest
Guest




MySQL question - Page 2 Empty
PostSubject: Re: MySQL question   MySQL question - Page 2 EmptyFri Sep 05, 2008 11:26 am

ibis wrote:
Kate P wrote:
ibis wrote
Quote :
You can also use (as I often do) Windows-IIS-mySQL-PHP, which makes one a WIMPy programmer.

This playing around with computer languages is fun afterall... Wink

Cheeky - I'll have you know you're using my intellectual property there, and modifying it without my permission. I coined WIMP back in 2001, during a training course I was running!

Mea culpa - my mistake, ibis was in assuming after your more than robust defence of the Lisbon treaty, that there could possibly be anything WIMPy about you. Laughing

Threads like this, on a more serious note, always make me conscious of how little I know about how the very machine I'm working on now, actually works. My secondary school was one of the first in the country to have a computer lab - of Amstrads /Commodore 64s. I learned Basic in second year and nothing since. But it is interesting to observe you and EVM conversing in a totally different language that to me just illustrates the pace of change in the computer industry in the last 20 years.
Back to top Go down
Guest
Guest




MySQL question - Page 2 Empty
PostSubject: Re: MySQL question   MySQL question - Page 2 EmptyFri Sep 05, 2008 2:09 pm

TheGeniusOfCork wrote:

I must really be a lamplighter, because I can add Perl, Python, Flash Graphics Design, Actionscript, ObjectC, Squeak SmallTalk, and of course the oldies, Fortran, C, C++, Rexx, Pascal, Basic, oh! and Blender the 3-d Graphic Design system, and of course Mathematica, Linpack,... oh. when will I stop bragging. Very Happy

I

Eh...never, you are from feckin' Cork after all! Wink
Back to top Go down
Guest
Guest




MySQL question - Page 2 Empty
PostSubject: Re: MySQL question   MySQL question - Page 2 EmptyFri Sep 05, 2008 8:15 pm

Ah the good old days - reading through my Oracle SQL reference book, debugging the query generation code, talking to my screen - simple pleasures.
Back to top Go down
Guest
Guest




MySQL question - Page 2 Empty
PostSubject: Re: MySQL question   MySQL question - Page 2 EmptyThu Sep 18, 2008 6:49 am

EvotingMachine0197 wrote:
Well then , coiner of WIMP, is it possible to leave a continuously running PHP script on a host server ? ie. if I make a script that say .. has to load 10000 website homepages in turn and search for a string - and if (string exists) {save url to file}
Yes you could do that but the more elegant approach would be to use the curl function to copy each homepage and then process the data in the background. You will find that of the 10K websites, some will no longer exist, some will have moved (301/302 results) and others may be forbidden/coming soon/pointed to a parking page. Normally the databases here check millions of websites when surveying TLDs such as ie/eu/co.uk etc (in addition to tracking over 210 million active and inactive domains (2000-2008)). You can run php scripts from the command line but you've got to constantly monitor scripts like this to make sure that they don't go wild and the way that some idiots write code, they can unexpectedly crash even the best written crawlers and spiders.

Quote :
This could take hours conceiveably (spell) ?
At least a few hours at least if you do it linearly. The timeouts and the missing websites will increase the time taken. The mistake that a lot of people seem to make is to try to process the data when the webpage collection program is running.

Quote :
Is it normal for a host server to put a time limit on execution time ?
It would be best to have a timeout inside the script rather than relying on the global script execution limit. With curl you can set that time out and I think that most of the curl functions are available in PHP.

If you need to read a book about this kind of stuff, the O'Reilly "Spidering Hacks" book is a good one though it is heavily reliant on Perl rather than PHP.

Regards...jmcc
Back to top Go down
Ex
Fourth Master: Growth
Ex


Number of posts : 4226
Registration date : 2008-03-11

MySQL question - Page 2 Empty
PostSubject: Re: MySQL question   MySQL question - Page 2 EmptyThu Sep 18, 2008 12:17 pm

Thanks jmcc. I have not used curl before so I checked it out on php.net. It does seem to have a fairly powerful function set in PHP.

Which leads nicely to something eles I wanted to do, which was to get my server to behave as a client and log into another website ??
This introduces the whole area of HTTP POST and cookies and all that stuff which I haven't got my head around yet.
Back to top Go down
Guest
Guest




MySQL question - Page 2 Empty
PostSubject: Re: MySQL question   MySQL question - Page 2 EmptyThu Sep 18, 2008 12:22 pm

EvotingMachine0197 wrote:
Thanks jmcc. I have not used curl before so I checked it out on php.net. It does seem to have a fairly powerful function set in PHP.

Which leads nicely to something eles I wanted to do, which was to get my server to behave as a client and log into another website ??
This introduces the whole area of HTTP POST and cookies and all that stuff which I haven't got my head around yet.

You really will need curl for that! It's certainly doable though - about ten lines of code.
Back to top Go down
Guest
Guest




MySQL question - Page 2 Empty
PostSubject: Re: MySQL question   MySQL question - Page 2 EmptyFri Sep 19, 2008 8:00 pm

There should be a few tutorials on using Curl and curl in php to log into sites. But the book I mentioned earlier is one of the best introductions that I've seen (there is not a lot of books on the subject as spider/crawler development tends to be a very specialised thing). Though as Ibis said above, it is only a few lines of code to get it working.

Another important thing is to be wary of how your script handles HTTP codes. Sometimes you can find that web admins create their own HTTP codes (0, 999 or some other weird code) and if your script is expecting a real result code, it may hang - hence the need for a timeout and a HTTP result code parser with a default exit code.

Regards...jmcc
Back to top Go down
Ex
Fourth Master: Growth
Ex


Number of posts : 4226
Registration date : 2008-03-11

MySQL question - Page 2 Empty
PostSubject: Re: MySQL question   MySQL question - Page 2 EmptyMon Jan 19, 2009 6:06 pm

I'm having trouble inserting strings containing Irish vowels áéíóú into a MySQL db.
If I submit the query directly, it's OK
If I import the query in a text file the fada vowels are not recognised. The get converted to question marks.

I've tried changing the character set a few times but no good.

Anyone ?
Back to top Go down
Guest
Guest




MySQL question - Page 2 Empty
PostSubject: Re: MySQL question   MySQL question - Page 2 EmptyMon Jan 19, 2009 6:23 pm

EvotingMachine0197 wrote:
I'm having trouble inserting strings containing Irish vowels áéíóú into a MySQL db.
If I submit the query directly, it's OK
If I import the query in a text file the fada vowels are not recognised. The get converted to question marks.

I've tried changing the character set a few times but no good.

Anyone ?

That looks like a character set issue - if you're using LOAD DATA INFILE to import have you tried adding "CHARACTER SET":

LOAD DATA INFILE 'file_name' INTO TABLE tbl_name CHARACTER SET charset_name

Could try 'utf8' as the character set. There's also a general setting "character_set_filesystem" for the server, which sets the default character set for file imports.
Back to top Go down
Sponsored content





MySQL question - Page 2 Empty
PostSubject: Re: MySQL question   MySQL question - Page 2 Empty

Back to top Go down
 
MySQL question
Back to top 
Page 2 of 2Go to page : Previous  1, 2
 Similar topics
-
» A Question of VISAs
» A question I never found the answer to.
» Cash and collateral: IL&P, ILA, TSB - a question
» The 64 Billion Euro question - FAQ
» Inflation / Banking / Interest Rate Question.

Permissions in this forum:You cannot reply to topics in this forum
Machine Nation  :: Machine Nation :: Science, Technology, Engineering, Computers-
Jump to: