These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

RETURNED - EVE Marketeer - Market Data Aggregation & Information

Author
Scrapyard Bob
EVE University
Ivy League
#41 - 2011-10-01 02:29:44 UTC
Callean Drevus wrote:

Also, I might need to start processing EMD orders in the same way I now process EMK orders, since that seems to be the fastest way, and I now get an immense backlog of EMD orders :P or I could just work on making order processing faster.


As in, making the upload tool send them directly to EMD instead of passing them through the EMK site? That would be good, at least EMD would get its updates even if EMK is having trouble (or vice-versa).

(Tonight, in order to update market data, I'm using the EMD market-scanner page, because the EMK page button to start scanning is not showing up.)
Callean Drevus
Perkone
Caldari State
#42 - 2011-10-01 07:30:18 UTC  |  Edited by: Callean Drevus
It actually does directly send them to EMD, even before it sends them to EMK. So in that regard everything plods along pretty well even if the website is down (though the tool will keep complaining about not being able to upload to EMK, it should have uploaded to EMD fine) :P

I'll take a look at the scanner page, maybe something to do with the move.

Developer/Creator of EVE Marketeer

Scrapyard Bob
EVE University
Ivy League
#43 - 2011-10-04 15:17:51 UTC
The scanner page is definitely still having issues, if it does come up, it eventually ends up with a gateway timeout error - which means my noble attempt to leave an alt logged in overnight comes to naught.

On the feature request side:

http://www.evemarketeer.com/statistics/uploads
- It would be better if this page was sorted by region name in alphabetical order, instead of by RegionID

http://www.evemarketeer.com/statistics/regions
- same request, please sort by region name

http://www.evemarketeer.com/item/itemprice/Oxygen_Isotopes
- On the regional prices tab, it would be useful if there was an indicator as to how current the data is for each region. Maybe presented as "The Forge (3 hrs)" or "Domain (2 days)"
- The orders tab, doesn't let me look at other regions
Scrapyard Bob
EVE University
Ivy League
#44 - 2011-10-04 15:24:03 UTC
One other question - does the EMK uploader send data to EVE Central as well?

(The fact that you send data to EMD in addition to EMK is a key reason I use your uploader.)
Callean Drevus
Perkone
Caldari State
#45 - 2011-10-04 18:32:00 UTC
Scrapyard Bob wrote:
The scanner page is definitely still having issues, if it does come up, it eventually ends up with a gateway timeout error - which means my noble attempt to leave an alt logged in overnight comes to naught.

On the feature request side:

http://www.evemarketeer.com/statistics/uploads
- It would be better if this page was sorted by region name in alphabetical order, instead of by RegionID

http://www.evemarketeer.com/statistics/regions
- same request, please sort by region name

http://www.evemarketeer.com/item/itemprice/Oxygen_Isotopes
- On the regional prices tab, it would be useful if there was an indicator as to how current the data is for each region. Maybe presented as "The Forge (3 hrs)" or "Domain (2 days)"
- The orders tab, doesn't let me look at other regions


Thank you for your feedback.

Sorting on the regions page makes sense, I'll fix that.

Sorting on the uploads page is already by name, only from left to right and up to down, instead of up to down left to right.

The itemprice page currently has a field that shows you how recent the data was. I agree it might be useful to add it to the regional tab.
I do wonder what you mean by, the orders tab does not let me look at other regions though. I think your issue is mainly with having to reload the entire page to switch regions, while it could be done faster? Or that you would like to quickly look at a different region without changing your default?

In regards to the uploader, it does not upload to EVE Central. Nor do I have plans to support it in the future. I do understand Muscaat of EVE Markets made an uploader that uploaded to all of our websites (EC, EMK and EMD), though the one that I can download from his website at this moment does not work (at least on my PC).

Lastly, the uploader page, I have no idea why it would be giving gateway timeout errors. I'll test it myself and see what I can dig up. It it not really acceptable that people leave their computers on all night to upload, only to find that it crashes halfway trough. Otherwise I'll just make the refresh AJAX too, and have it automagically refresh it's list if it detects no activity.

Developer/Creator of EVE Marketeer

Callean Drevus
Perkone
Caldari State
#46 - 2011-10-13 22:06:33 UTC
Did some minor fixes, and implemented caching today (for home and for statistics pages). Caching might be a bit erratic at the moment.

Developer/Creator of EVE Marketeer

Domi Etymology
Violent Alternatives
#47 - 2011-10-19 00:11:44 UTC
getting an error when trying to log in at the moment, and this error message

Server Time: 2011-10-19 00:10:43 Warning: Invalid argument supplied for foreach() in /var/www/core/lib/dwoo/plugins/personal/region_list.php on line 28
Turhan Bey
EVE University
Ivy League
#48 - 2011-10-19 10:08:10 UTC
Get well soon! Smile
Controller Vrelk
The Scope
Gallente Federation
#49 - 2011-10-19 17:18:09 UTC  |  Edited by: Controller Vrelk
When you're feeling better.

If you don't get better soon enough you will wish you hadn't red this due to how quickly I keep adding things to this list, but get better anyway.

And now for the long list of bugs/suggestions... (I did put these in the feedback system on the site as well)

1)
http://www.evemarketeer.com/player/account
Warning: implode(): Invalid arguments passed in /var/www/mod/player.php on line 75

2)
You also might want to turn the order history page when looking at an item into a paged list, or shorten it. Right now it takes a long time to load and is very long.

3)
Shorten/page the upload history page.
Also some of mine are showing -1 for the credits earned.
ex: Assault Missile Launcher II, Assault Missile Launcher I Blueprint, Ship Scanner II

4)
On the mission statistics page, it says I need to add a character to the account, when I have and it has pulled the API for it. Maybe say what permissions from the API are needed when showing that error.

5)
Same error as above but on the My Transactions page. I have market orders and wallet transactions enabled on the API.

6)
What do the red/green items on the API page mean (color wise)? If it means it is enabled on the API, it is wrong. Such as CalendarEventAttendees is green but I don't have it enabled for that API key, and WalletTransactions is red yet I have it enabled on the API (see #5, maybe I should have made this a flowchart =] ).

7)
On the API page all of the titles say they are for the corporation when I entered a character API key

8)
On the Profit per Ship page, the titles for the columns should me reworded. Exactly what is a mrokh and mhulk? I would have assumed it meant meta, but miners/strip miners don't have the meta versions of them. I can see a deep-core Rokh if thats what it meant.
Callean Drevus
Perkone
Caldari State
#50 - 2011-10-19 18:35:12 UTC
Thank you for your kindness Blink I'm still feeling unwell though.

Which is one of the reasons I'm uploading the entire EVE Marketeer codebase to Github. Previously, I already added a few people to the mercurial repository I was using before, but this seems immensely easier. Besides, everyone will be able to contribute instead of just a few.

I'll also expose access to the live database to a few people that I trust not to give away private information.

This will hopefully allow updates to happen even if I'm not feeling ok Roll.

The only thing I'm struggling with is providing a copy of the database. I could allow a complete download, but at the moment it's about 45 GB, which is probably a tad much for most people. Anyone have any ideas?

Developer/Creator of EVE Marketeer

Controller Vrelk
The Scope
Gallente Federation
#51 - 2011-10-19 20:28:57 UTC
Callean Drevus wrote:
Thank you for your kindness Blink I'm still feeling unwell though.

Which is one of the reasons I'm uploading the entire EVE Marketeer codebase to Github. Previously, I already added a few people to the mercurial repository I was using before, but this seems immensely easier. Besides, everyone will be able to contribute instead of just a few.

I'll also expose access to the live database to a few people that I trust not to give away private information.

This will hopefully allow updates to happen even if I'm not feeling ok Roll.

The only thing I'm struggling with is providing a copy of the database. I could allow a complete download, but at the moment it's about 45 GB, which is probably a tad much for most people. Anyone have any ideas?


Maybe just include active orders, or just ones that have been seen in the past x days in the download.
Callean Drevus
Perkone
Caldari State
#52 - 2011-10-19 21:24:32 UTC
Hmm, git is being recalcitrant as usual. I wonder why I still try... Maybe just stay with mercurial...

Developer/Creator of EVE Marketeer

Controller Vrelk
The Scope
Gallente Federation
#53 - 2011-10-19 21:30:38 UTC  |  Edited by: Controller Vrelk
Callean Drevus wrote:
Hmm, git is being recalcitrant as usual. I wonder why I still try... Maybe just stay with mercurial...


I don't remember what it was but with another project that I think was on google code, people could upload patches that the devs would then incorporate into the main code if they thought it was good. I don't remember what system it was with, but maybe that would be helpful allowing anyone to submit code patches to you.

It seems I about doubled/tripled the length of the feedback page on the site now =]


Edit: I found what I was looking for.
I don't know if github has something like this or not, but it might be helpful for patches.
http://code.google.com/p/hlstatsxcommunity/source/browse/web/templates/default/thanksto.tpl?edit=1
So basically you edit the code on the site, describe the change and you can suggest the patch to the devs.


On a side note, look into adding this link to your sig for the people that don't have crome. http://userscripts.org/scripts/show/112772
Callean Drevus
Perkone
Caldari State
#54 - 2011-10-19 21:51:51 UTC
Woot! EVE Marketeer was on github. And then I noticed I had some production passwords in the source that should have been hidden. Oh well, I'll give up for today. Tomorrow will probably see me simply committing the damn thing to bitbucket. I do (again) not feel like dealing with git ever again.

And yes, both of those websites allow people to commit patches which I can then merge into the main codebase. That's kind of the point Blink

You won't be able to edit the code on the site itself though, but using mercurial should be relatively painless (as opposed to git).

EDIT: Regarding my sig, yeah, but I seem to not be able to edit it anymore.

Developer/Creator of EVE Marketeer

Controller Vrelk
The Scope
Gallente Federation
#55 - 2011-10-19 21:58:51 UTC  |  Edited by: Controller Vrelk
Callean Drevus wrote:
Woot! EVE Marketeer was on github. And then I noticed I had some production passwords in the source that should have been hidden. Oh well, I'll give up for today. Tomorrow will probably see me simply committing the damn thing to bitbucket. I do (again) not feel like dealing with git ever again.

And yes, both of those websites allow people to commit patches which I can then merge into the main codebase. That's kind of the point Blink

You won't be able to edit the code on the site itself though, but using mercurial should be relatively painless (as opposed to git).

EDIT: Regarding my sig, yeah, but I seem to not be able to edit it anymore.


I haven't used github much. I'm used to SVN. I'm assuming you will post a link to the repo as soon as you are done setting it up?


Edit: just curious. What are the machine specs this is run off of? I know it's ubuntu and running nginx/apache.


Edit 2: I have noticed things loading slowly at times. So maybe add the page compile/load time and the number of sql queries at the bottom of every page. Something that might help us bug reporters to find things that are an issue. (Especially helpful once you release the code).
Callean Drevus
Perkone
Caldari State
#56 - 2011-10-20 06:41:43 UTC
Try appending ?debug=3 to any URL if you want to see loadtime and queries.

The specs are:
i3 Intel processor
4 Gb RAM
2 1 Gb SATA disks in raid 1
100 Mbit uplink

Those are probably the most relevant ;)

Developer/Creator of EVE Marketeer

Controller Vrelk
The Scope
Gallente Federation
#57 - 2011-10-20 16:22:41 UTC  |  Edited by: Controller Vrelk
Callean Drevus wrote:
Try appending ?debug=3 to any URL if you want to see loadtime and queries.

The specs are:
i3 Intel processor
4 Gb RAM
2 1 Gb SATA disks in raid 1
100 Mbit uplink

Those are probably the most relevant ;)


Ok. I was thinking of something more along the lines of it showing all the time in a way that doesn't mess up the page layout.
ex. http://chubaka.dyndns.biz/~ci/ipb_stats.png
Callean Drevus
Perkone
Caldari State
#58 - 2011-10-20 16:29:24 UTC
I know you were :) this is just the best it can do at the moment. Havent found a decent way to incorporate the stats into the page yet.

Developer/Creator of EVE Marketeer

Controller Vrelk
The Scope
Gallente Federation
#59 - 2011-10-20 17:00:37 UTC  |  Edited by: Controller Vrelk
Callean Drevus wrote:
I know you were :) this is just the best it can do at the moment. Havent found a decent way to incorporate the stats into the page yet.

Ok. I thought the screenshot would help give you an idea of what I meant and an idea for a way to incorporate it into the site.

How about this?
http://chubaka.dyndns.biz/~ci/evemarketeer_stats.png
(I just grabbed the code from the IPB Forums site and stuck it in there using your theme.)
http://chubaka.dyndns.biz/~ci/stats_code.txt (if you like it, there is the code/links to images. I think they are the famfamfam silk icon ones though)
Callean Drevus
Perkone
Caldari State
#60 - 2011-10-20 17:28:20 UTC
Lol, nice going there Blink

Don't you think it's a lot of space (distraction) for just the stats though? That has been what has kept me from implementing it this way. I would do it if I had something else to show in that bar, but right now, it's just a waste of space (though I doubt many people would mind, spending 10% of the (visible) page height on stats feels wrong to me)

That said, think up something useful to go on the left side of that box, and it would be ok ;) maybe we should also show number of users active or something P

Developer/Creator of EVE Marketeer