These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

RETURNED - EVE Marketeer - Market Data Aggregation & Information

Author
Scrapyard Bob
EVE University
Ivy League
#121 - 2011-11-19 00:16:27 UTC
If I make an API query call with a wrong typeID like:

http://www.evemarketeer.com/api/info/37_40_36_0_11399_38_35_34_39/xml/10000030/sell_lowest5
or
http://www.evemarketeer.com/api/info/37_40_36__11399_38_35_34_39/xml/10000030/sell_lowest5

I should still get back 9 rows. Instead, I only get back 8 rows in the above example. Ideally, that 4th row should simply have a zero in the sell_lowest5 and the datetime should be a date from a year ago (or the start of the epoch?).
Turhan Bey
EVE University
Ivy League
#122 - 2011-11-19 17:04:16 UTC
Jita Prices API call is returning some very outdated information. (I've only tried that call, not sure if other API calls are functioning well or not.) Example for Mechanical Parts:

http://www.evemarketeer.com/api/jita_info/3689/xml

Select output, seems to reflect the state of the market during the datetime in question, but the datetime is rather old:

type_id: 3689
region_id: 10000002
station_id: 60003760
datetime: 2011-10-20 18:09:30

sell_lowest: 11239.98
sell_highest: 34000
sell_lowest5: 11243.358326225
buy_highest: 8501.04
buy_avg: 7665.2050780545
buy_lowest: 8501.04
buy_highest5: 8501.01826388197
Callean Drevus
Perkone
Caldari State
#123 - 2011-11-20 09:54:10 UTC
Huh? I replied to this yesterday...

In any case, will be fixed (the outdated information). And will be taken into consideration (the API queries), ideally you just wouldn't call the API with invalid information Blink

Developer/Creator of EVE Marketeer

Scrapyard Bob
EVE University
Ivy League
#124 - 2011-11-20 13:28:54 UTC
Callean Drevus wrote:

In any case, will be fixed (the outdated information). And will be taken into consideration (the API queries), ideally you just wouldn't call the API with invalid information Blink


Blink

I live to break things. In this particular case, I was setting up a sheet to track T1 ship construction prices and estimated margins. Since I don't have typeIDs for the new BCs yet, I had to just either leave those cells blank or put zeros in.

Zeros as a "invalid" return value are good, because you can easily exclude them with a CountIf() in most spreadsheets. And they don't screw up any Sum() functions. So if I pull 3 regions, I just do the calculation for the average price column as:

AvgPrice = Sum(A4:C4) / CountIf(A4:C4,">0")
Callean Drevus
Perkone
Caldari State
#125 - 2011-11-20 14:16:06 UTC
Ok, well, since you ARE our best uploader, I think I might just make it accept invalid numbers as zero rows. Will see what I can do P

Developer/Creator of EVE Marketeer

Callean Drevus
Perkone
Caldari State
#126 - 2011-11-22 20:48:17 UTC
It is monstrous what sort of an effect a specific server can have on your application. EVE marketeer is now running on hardware 4 times worse than the previous setup, but it's processing way faster than ever before... This could be awesome :D

First impressions of Storm on Demand are very good.

Developer/Creator of EVE Marketeer

Scrapyard Bob
EVE University
Ivy League
#127 - 2011-11-23 05:56:33 UTC
Tried using the new uploader again tonight, went back to EMD's. Hopefully you can pull the data from EMD Blink

The issue that I have with the EMK uploader is that it's not very responsive to UI input when it's waiting for communication to the various websites to complete. I'm not sure what the fix there is, other then to split it into multiple threads, each thread uploading to a different site so that a single site being slow about processing doesn't impact the upload rates of the other sites.
Turhan Bey
EVE University
Ivy League
#128 - 2011-11-23 21:13:00 UTC
Error return for current report bug, just in case it is helpful to you. :)

Call was:
http://www.evemarketeer.com/api/jita_info/34/xml

Error returned:
Fatal error: Uncaught exception 'Dwoo_Exception' with message 'Dwoo->get/Dwoo->output's first argument must be a Dwoo_ITemplate (i.e. Dwoo_Template_File) or a valid path to a template file' in /var/www/www.evemarketeer.com/core/lib/dwoo/Dwoo.php:314 Stack trace: #0 /var/www/www.evemarketeer.com/core/mod/template.php(33): Dwoo->get('core/tpl/api/ji...', Array) #1 /var/www/www.evemarketeer.com/core/mod/page.php(33): template_parse('api/jita_info.t...', Object(Request)) #2 /var/www/www.evemarketeer.com/core/main.php(69): page->renderBody(Object(Request)) #3 /var/www/www.evemarketeer.com/index.php(8): include('/var/www/www.ev...') #4 {main} thrown in /var/www/www.evemarketeer.com/core/lib/dwoo/Dwoo.php on line 314
Domi Etymology
Violent Alternatives
#129 - 2011-11-23 22:15:39 UTC
Scrapyard Bob wrote:
Tried using the new uploader again tonight, went back to EMD's. Hopefully you can pull the data from EMD Blink

The issue that I have with the EMK uploader is that it's not very responsive to UI input


I've been noticing this myself. It didn't actually click that it the headaches I've been having with the uploader were occuring while it was waiting for communication to the different websites. Over the last couple of days, I've turned off uploading to eve-central, which helped a ton. Also, whatever you did with your server upgrade increased responsiveness a ton :)

Currently, if I leave 3 clients uploading data simultaneously for 15 minutes, I'll come back and the uploader window has just frozen. If I stop the eve clients from checking data, the client *might* eventually unfreeze and the log will scroll like crazy until it's caught up. Scrapyard Bob's suggestion sounds spot on, split the upload functions into different threads, so that the delay in waiting for responses from uploads isn't sending the whole uploader program to **** :)
Scrapyard Bob
EVE University
Ivy League
#130 - 2011-11-26 06:29:04 UTC
Got one tonight while looking up an item:

http://www.evemarketeer.com/item/itemprice/Memory_Augmentation_-_Improved

Fatal error: Uncaught exception 'Dwoo_Exception' with message 'Dwoo->get/Dwoo->output's first argument must be a Dwoo_ITemplate (i.e. Dwoo_Template_File) or a valid path to a template file' in /var/www/www.evemarketeer.com/core/lib/dwoo/Dwoo.php:314 Stack trace: #0 /var/www/www.evemarketeer.com/core/mod/template.php(33): Dwoo->get('core/tpl/item/i...', Array) #1 /var/www/www.evemarketeer.com/core/mod/page.php(33): template_parse('item/itemprice....', Object(Request)) #2 /var/www/www.evemarketeer.com/core/main.php(69): page->renderBody(Object(Request)) #3 /var/www/www.evemarketeer.com/index.php(8): include('/var/www/www.ev...') #4 {main} thrown in /var/www/www.evemarketeer.com/core/lib/dwoo/Dwoo.php on line 314
Scrapyard Bob
EVE University
Ivy League
#131 - 2011-11-26 06:30:25 UTC
Domi Etymology wrote:

I've been noticing this myself. It didn't actually click that it the headaches I've been having with the uploader were occuring while it was waiting for communication to the different websites. Over the last couple of days, I've turned off uploading to eve-central, which helped a ton. Also, whatever you did with your server upgrade increased responsiveness a ton :)

Currently, if I leave 3 clients uploading data simultaneously for 15 minutes, I'll come back and the uploader window has just frozen. If I stop the eve clients from checking data, the client *might* eventually unfreeze and the log will scroll like crazy until it's caught up. Scrapyard Bob's suggestion sounds spot on, split the upload functions into different threads, so that the delay in waiting for responses from uploads isn't sending the whole uploader program to **** :)


I'm now running both upload tools at the same time - and letting them fight over the cache files. If EMK's uploader gets side-tracked and starts timing out, the EMD upload tool picks up the slack.
Callean Drevus
Perkone
Caldari State
#132 - 2011-11-27 00:27:04 UTC

@Scrapyard: You are correct. The uploader is indeed very unresponsive while uploading. I'd been thinking about that problem myself, but have obviously not found a fix for it yet. Having the uploads handled in different threads would be nice (and very effective, at least in terms of UI responsiveness), but I so far have been unable to implement it.

@Turhan Bey: Thank you. Besided the old data issue, which I'm trying to fix in the station_info API (and then link jita_info to that), it now seems to be giving a template error (which is weird, as those functions should not look at templates anyway).

Developer/Creator of EVE Marketeer

Callean Drevus
Perkone
Caldari State
#133 - 2011-11-27 11:34:00 UTC
Ok, I think I fixed station information now (and thereby the jita_info API too). I also fixed a bug in the 'my orders' page which caused it to not be able to scan your orders.

Developer/Creator of EVE Marketeer

Controller Vrelk
The Scope
Gallente Federation
#134 - 2011-11-30 01:13:30 UTC  |  Edited by: Controller Vrelk
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 80 bytes) in /var/www/www.evemarketeer.com/core/mod/content/yapeal.php on line 107

While trying to load the history for Tritanium


Also


The "Total Bounties" and "Total Bounty Taxes" always shows the total amount regardless of what you set the time to. If you set it to 1 day, it should show totals for the past day. I don't know about the mission totals as I haven't run any in the time the site was able to grab logs for.
Callean Drevus
Perkone
Caldari State
#135 - 2011-11-30 07:52:49 UTC
Lol, ok, pagination here I come. It seems the total history for one item finally exceeds the limits of php.

And yes, I'll look into the mission page, it seems it should react to the time limit, but I'm not sure about it. Today is the first time in a while I finally have some time to actually do something again, so I hope to do some cool stuff today.

Developer/Creator of EVE Marketeer

Controller Vrelk
The Scope
Gallente Federation
#136 - 2011-11-30 22:43:29 UTC  |  Edited by: Controller Vrelk
1) Maybe limit the history tab to 90 orders per page, or 30 and go by months rather than a year.

2) After waiting about 30 seconds for the orders tab to load:

Warning: file_put_contents(var//classpath.cache.d15.php): failed to open stream: Permission denied in /var/www/www.evemarketeer.com/core/lib/dwoo/Dwoo/Loader.php on line 86 Fatal error: Uncaught exception 'Dwoo_Exception' with message 'Could not write into var//classpath.cache.d15.php, either because the folder is not there (create it) or because of the chmod configuration (please ensure this directory is writable by php), alternatively you can change the directory used with $dwoo->setCompileDir() or provide a custom loader object with $dwoo->setLoader()' in /var/www/www.evemarketeer.com/core/lib/dwoo/Dwoo/Loader.php:87 Stack trace: #0 /var/www/www.evemarketeer.com/core/lib/dwoo/Dwoo/Loader.php(104): Dwoo_Loader->rebuildClassPathCache('/var/www/www.ev...', 'var//classpath....') #1 /var/www/www.evemarketeer.com/core/lib/dwoo/Dwoo/Compiler.php(2703): Dwoo_Loader->loadPlugin('stationname', true) #2 /var/www/www.evemarketeer.com/core/lib/dwoo/Dwoo/Compiler.php(2494): Dwoo_Compiler->getPluginType('stationname') #3 /var/www/www.evemarketeer.com/core/lib/dwoo/Dwoo/Compiler.php(1981): Dwoo_Compiler->replaceModifiers(Array, 'var', 12) #4 /var/www/www.evemarketeer.com/core/lib/dwoo/Dwoo/Compiler.php(1 in /var/www/www.evemarketeer.com/core/lib/dwoo/Dwoo/Loader.php on line 87

I don't know if this is because you are still working on it, or something like the history page was doing yesterday (loading a long time then giving an error).


3) Make sure you mark the things as done on the feedback page, you missed pagination, clicking on search result to submit, etc.

4) The checkbox for the signature won't save, it always stays empty and the sig stays private.

4) I like the new tables at the top of the items page and the other layout/design/css changes.

5) The uploader is screwed up. It stopped uploading at 301, then stopped, no errors, just stopped. Restarting it didn't fix it, nor did disabling all but EMK.

6) The uploads aren't being processed?
Controller Vrelk
The Scope
Gallente Federation
#137 - 2011-12-01 03:40:51 UTC  |  Edited by: Controller Vrelk
Alright, I see you made it show the totals for the time set, sorta. I say sorta because I noticed a counting problem.

http://compu.redshift-gaming.net/images/js/EMK_mission.png
Just noticed the offset in the numbering, oh well, you get the point.
Callean Drevus
Perkone
Caldari State
#138 - 2011-12-01 21:28:32 UTC
Most of those things fixed. And put the server in temporary reinforced mode while I mess with the database.

Developer/Creator of EVE Marketeer

Callean Drevus
Perkone
Caldari State
#139 - 2011-12-04 16:37:40 UTC  |  Edited by: Callean Drevus
Ok, so today I built a few of the features that have been on the wishlist for a while.

In addition, I also updated the database to use the new Crucible database.

The new things are:
- Show a aggregate of the stats for the top 5 regions. This means the website takes an average of the values (for example lowest sale price) in the 5 regions with the most orders. This should help counter any market manipulation attempts a bit. An API for this feature is coming soon.
- Allow filtering by station in the orders list, as well as clean the list up a bit by hiding most of the orders you might not want to see.
- Add an addition row to item information that shows the changes in price the past day, and also shows an estimate of the buy/sell ratio that is used in some computation on the website
- Importing from EVE Central and EVE Marketdata have been enabled again and do not seem to be giving any problems, except that those two websites are not sending a lot of data (not any data atm, which I cannot understand).

You might've noticed I've been working hard on fixing stuff that was on the website for a while. This is because I felt that the overal quality of some things on the website was a bit low. New features have been moved to the background a bit, but are certainly not forgotten.

Developer/Creator of EVE Marketeer

Controller Vrelk
The Scope
Gallente Federation
#140 - 2011-12-04 17:52:06 UTC
On the bounty tracker page:
Warning: Invalid argument supplied for foreach() in /var/www/www.evemarketeer.com/mod/missions.php on line 40

And the table is very screwed up now, only numbers where the names should be and all 0's in the amount column.

And it would appear that the totals are still wrong.