These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Crest -> MySql via PHP

Author
Risov Cobon-Han
Monte Inc
#1 - 2016-02-16 12:47:08 UTC
Hi,

Very inexperienced programmer here (as in not at experienced to the point of the only php I know I have learned by building some basic EVE related things)

Looking for some help with understanding whether or not I have gone about this in the correct way.
I'm trying to get the current lowest Jita sell price for a list of items and store that in a mysql table.

My code works... but I'm hoping to learn how to make it not obviously ridiculous, Here's how it currently goes:

Read in a list of typeids
For the first typeid, fetch the sell prices from public crest
Write all of the results back to a temporary table
Read in just the lowest price just for the required station from that table
Write that low price to a different table
Repeat 2 through 5 for next typeid

Full actual php is here http://pastebin.com/UPp1iKrZ

Currently takes 20+ seconds to work through typeids '34,35,36,37' for Jita - I'm looking to scale up to several hundred items for several stations, so the primary problem I think I have is with the speed of it all - just I'm not sure how to identify where it is specifically slow.
My thinking is that there should be a way to grab just the lowest price just for the wanted station which would cut out two whole steps, but I can't work that out.

Any help appreciated
Princess Honneamise
#2 - 2016-02-16 14:44:44 UTC


1)
from my point of view it is useless to use a temporary table .
you can get the lowest order directly in your for loop .

2)
file_get_contents() is a blocking call.
if you have tons of request to do why not use threads ?

EVE MOONS PROJECT

http://eve-moons.com

The EVE MOONS PROJECT is the most complete and accurate moons database for Eve Online

Dragonaire
Here there be Dragons
#3 - 2016-02-17 05:27:32 UTC
I've done a quick pass re-write on your code which you can view at:
http://pastebin.com/4XxhJ8hp

I don't know if it'll fix the issues you are having but it should help anyway Smile

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Risov Cobon-Han
Monte Inc
#4 - 2016-02-17 10:33:46 UTC
Thanks for the help, have managed to double the speed and eliminate the excessive temporary database writes and reads.

@Dragonaire, many thanks for taking the time to work through and mark up all those edits!
Have already used some of the suggested edits from your tidy up pass, and will incorporate several more when I get to a final version.

@Princess
1 - Yup - temporary table being useless was always my thinking, I was just having a stack of trouble working out how Not to need it.
Still haven't managed to get the lowest order within that same loop. But, after a lot of false starts, did manage a workaround that while probably still horribly inefficient is much better than what I had.
- found I couldn't work out how to either exclude other stations, or find the lowest price (much, much pain trying to understand just how a multidimensional array actually works Ugh ). Instead was able to identify if an array value was at the correct station, and then push just the price out into a new array, and from there I was able to sort that array and then use the first value.

2 - I have no idea how to get at that json data without using file_get_contents - will be my google project for tomorrow.
Equally so, no idea how to use multiple threads - will try and work on that after replacing file_get

For those playing along at home, shrug, maybe this is useful to someone else who wants to build something similar who coincidentally is just as unknowledge'd as I -> Here's how it currently all looks: http://pastebin.com/dWExLjwC
(Not suggesting that anyone actually uses this; Plenty to be done to it still... but... well, it works as it currently is if your standards aren't exceptionally high Lol ...)
Dragonaire
Here there be Dragons
#5 - 2016-02-18 09:06:16 UTC
Here's another version of your updated code with some best practices / other enhancements to it. Should be faster as well since it's finding the lowest price directly without sort or additional loop Smile

http://pastebin.com/umPc59nQ

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Risov Cobon-Han
Monte Inc
#6 - 2016-02-19 13:13:31 UTC
Thanks once again for the edits, notes and the more or less complete re-write to make it all better.
Performance is vastly improved from my initial version, and I was thinking I would need to spend some weekend time sorting out how and what to do if values didn't exist or weren't returned, so thanks also for including all of the error handling.

A random good Samaritan (who I presume wishes to remain anonymous by virtue of his contact) left me a mail with just a link (and nothing more) to a decade old blogpost about using curl to fetch data, from which I was able to get a working function that I've included as get_data - Some testing has shown that using this method rather than the previous file_get_contents method saves a fraction of a second per query, so that's why that is there now. (Whilst also ticking off Princess Honneamise suggestion to not use file_get_contents)

The crest/json/'array within an array of arrays' format broke your $lowest, min bit - Wasn't correctly identifying the locationID and thus was storing the INT_MAX for everything. Managed to find the problem;
$item['location'] is itself an array, but moving over to $item['location']['id'] actually gets to the station to then match back to locationID, so that's why that change is now in there.

Overall is taking 35 seconds to complete a run through with 20 mostly random items in the itemlist table - so working on understanding and implementing some form of threading will be my next challenge.

Current version updated here:
http://pastebin.com/aXj0kMA7