These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

RETURNED - EVE Marketeer - Market Data Aggregation & Information

Author
Vaerah Vahrokha
Vahrokh Consulting
#541 - 2012-04-14 00:37:27 UTC  |  Edited by: Vaerah Vahrokha
Website completely dies in US prime time. Some times it gives some message like: "too many connections" other times like today it just goes with a long pause then "connection refused", "connection timeout" and similar.
Dalden V
Yellow Lounge Industries
#542 - 2012-04-14 12:54:17 UTC
Vaerah Vahrokha wrote:
Website completely dies in US prime time. Some times it gives some message like: "too many connections" other times like today it just goes with a long pause then "connection refused", "connection timeout" and similar.


Thanks Vaerah. I was wondering if it was just me that cannot access the site :). Today I'm constantly getting connection timeouts.
plan q
Perkone
Caldari State
#543 - 2012-04-14 14:22:31 UTC  |  Edited by: plan q
ah pretty sure it has been down since
2012/04/13 20:35:43
no api calls since then :D\
--
and from twitter stalking it looks like his vps is down
Callean Drevus
Perkone
Caldari State
#544 - 2012-04-14 15:09:09 UTC  |  Edited by: Callean Drevus
Balls, it seems the site either crashed, or someone at my datacenter did something incredibly stupid, since my server died (presumably) without reason some 14 hours ago. I'm currently investigating this issue.

UPDATE: Oh joy, the server is moving, and my hosting company does not know why...

Developer/Creator of EVE Marketeer

LifeHatesMe
LifeHatesUsAll
#545 - 2012-04-14 15:49:24 UTC  |  Edited by: LifeHatesMe
-snip- accidental post
LifeHatesMe
LifeHatesUsAll
#546 - 2012-04-14 15:49:48 UTC  |  Edited by: LifeHatesMe
Callean Drevus wrote:

How about this?
http://chubaka.dyndns.biz/~ci/evemarketeer_stats.png
(I just grabbed the code from the IPB Forums site and stuck it in there using your theme.)
http://chubaka.dyndns.biz/~ci/stats_code.txt
(if you like it, there is the code/links to images. I think they are the famfamfam silk icon ones though)



Your links are broken;

* dig chubaka.dyndns.biz +short
35.32.132.87
* ping -c 1 35.32.132.87
PING 35.32.132.87 (35.32.132.87) 56(84) bytes of data.
--- 35.32.132.87 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 10000ms

Sounds like you need to fixy your dynamic dns service? :)
Callean Drevus
Perkone
Caldari State
#547 - 2012-04-14 16:45:59 UTC
I didn't really understand what you meant to say...

The links you provided link only to a 404 page not found.

In regards to the server being down. The sudden move of my server was cancelled, but it doesn't seem like that will bring it up again quickly, so I can only ask you to bear with it for the moment.

Developer/Creator of EVE Marketeer

Zifrian
The Frog Pond
Ribbit.
#548 - 2012-04-14 19:10:12 UTC
I'm sure it's all related to your current posts, but the following address: http://www.evemarketeer.com/api/averages/10000002/30/620/xml

Gave the following error: "Unable to connect to the remote server

Not sure if that helps

Thanks for looking into it.

Maximze your Industry Potential! - Download EVE Isk per Hour!

Import CCP's SDE - EVE SDE Database Builder

LifeHatesMe
LifeHatesUsAll
#549 - 2012-04-14 19:13:25 UTC  |  Edited by: LifeHatesMe
Callean Drevus wrote:
I didn't really understand what you meant to say...

The links you provided link only to a 404 page not found.

In regards to the server being down. The sudden move of my server was cancelled, but it doesn't seem like that will bring it up again quickly, so I can only ask you to bear with it for the moment.
Something is really wrong with your routing mister. It looks like some sort of networking issue :(

Edit: Are you hosting your website on the same server that's doing DNS services?
Desmont McCallock
#550 - 2012-04-14 20:34:58 UTC
Do you all fail on reading that his server is down?
Aineko Macx
#551 - 2012-04-15 15:57:47 UTC  |  Edited by: Aineko Macx
- When doing a history query of multiple items, the API is only returning data for the first typeID specified, ignoring the others

EDIT:
- If the DB has no data for the item or no data for the specified time range, the history query throws a warning:
Warning: get_object_vars() expects parameter 1 to be object, string given in /var/www/www.evemarketeer.com/mod/api.php on line 389
Warning: Invalid argument supplied for foreach() in /var/www/www.evemarketeer.com/mod/api.php on line 390
LifeHatesMe
LifeHatesUsAll
#552 - 2012-04-15 19:15:50 UTC  |  Edited by: LifeHatesMe
Desmont McCallock wrote:
Do you all fail on reading that his server is down?
I can read.

dig www.evemarketeer.com ns
;; Got bad packet: extra input data
101 bytes
2d 0f 81 80 00 01 00 00 00 01 00 00 03 77 77 77 -............www
0c 65 76 65 6d 61 72 6b 65 74 65 65 72 03 63 6f .evemarketeer.co
6d 00 00 02 00 01 c0 10 00 06 00 01 00 00 38 40 m.............8@
00 33 00 00 0a 68 6f 73 74 6d 61 73 74 65 72 0c .3...hostmaster.
65 76 65 6d 61 72 6b 65 74 65 65 72 03 63 6f 6d evemarketeer.com
00 77 df 4d 38 00 00 38 40 00 00 0e 10 00 12 75 .w.M8..8@......u
00 00 00 38 40 ...8@

^ Get a better DNS resolver?


dig www.evemarketeer.com ns @ns1.coronum.nl +short

; <<>> DiG 9.7.3 <<>> www.evemarketeer.com ns @ns1.coronum.nl +short
;; global options: +cmd
;; connection timed out; no servers could be reached


dig evemarketeer.com a +short
89.31.102.172 (redirects)
dig www.evemarketeer.com a +short
50.28.89.81 (site loads)

Why not change the @ DNS to point to the new ip address? I think you should get a better name server resolution, there are several out there. I'd suggest; http://zoneedit.com

Or failing that.. http://freedns.afraid.org/ :)

Edit: Looks like your on liquid web;
http://www.liquidweb.com/dedicated/single-processor.html

Why not switch to a more reliable VPS provider like linode? Just curious.
Vaerah Vahrokha
Vahrokh Consulting
#553 - 2012-04-17 00:35:37 UTC
While accessing the website I am getting blank pages that say:

502 Bad Gateway

nginx/0.7.65

When it does not do that, it takes 30 seconds to show a page (hilariously, it says the query took 0.8 secs to run).
LifeHatesMe
LifeHatesUsAll
#554 - 2012-04-17 01:04:13 UTC
Vaerah Vahrokha wrote:
While accessing the website I am getting blank pages that say:

502 Bad Gateway

nginx/0.7.65

When it does not do that, it takes 30 seconds to show a page (hilariously, it says the query took 0.8 secs to run).
NGINX is a lightweight http daemon service.

Bad Gateway seems to mean that how it's connected with the PHP parser is not syncing properly, there are many reasons this happens. I don't know your config, are you using nginx for everything? A reverse proxy with apache? or something else?

I still think you could get a better host. The whole reason people use nginx in the first place is because they lack enough ram to use Apache reliably. Question
Philderbeast
Native Freshfood
Minmatar Republic
#555 - 2012-04-17 05:14:18 UTC
LifeHatesMe wrote:
Vaerah Vahrokha wrote:
While accessing the website I am getting blank pages that say:

502 Bad Gateway

nginx/0.7.65

When it does not do that, it takes 30 seconds to show a page (hilariously, it says the query took 0.8 secs to run).
NGINX is a lightweight http daemon service.

Bad Gateway seems to mean that how it's connected with the PHP parser is not syncing properly, there are many reasons this happens. I don't know your config, are you using nginx for everything? A reverse proxy with apache? or something else?

I still think you could get a better host. The whole reason people use nginx in the first place is because they lack enough ram to use Apache reliably. Question


not so much they lack the ram, but nginx is faster at serving static content then apache and scales up much better then apache. infact many large sites use it like face book as it scales much better.
Scrapyard Bob
EVE University
Ivy League
#556 - 2012-04-19 17:32:01 UTC
Prices are now about 7 days old according to the little green/yellow/orange dots.

http://www.evemarketeer.com/item/group_summary/18

There's also close to 100k in queued uploads.
Controller Vrelk
The Scope
Gallente Federation
#557 - 2012-04-20 01:45:42 UTC
LifeHatesMe wrote:
Callean Drevus wrote:

How about this?
http://chubaka.dyndns.biz/~ci/evemarketeer_stats.png
(I just grabbed the code from the IPB Forums site and stuck it in there using your theme.)
http://chubaka.dyndns.biz/~ci/stats_code.txt
(if you like it, there is the code/links to images. I think they are the famfamfam silk icon ones though)



Your links are broken;

* dig chubaka.dyndns.biz +short
35.32.132.87
* ping -c 1 35.32.132.87
PING 35.32.132.87 (35.32.132.87) 56(84) bytes of data.
--- 35.32.132.87 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 10000ms

Sounds like you need to fixy your dynamic dns service? :)

That is from a long time ago. Well, that used to exist. It was on a computer sitting in my dorm room. As for the packet loss, my router is set not to respond to pings.
Callean Drevus
Perkone
Caldari State
#558 - 2012-04-21 20:21:43 UTC  |  Edited by: Callean Drevus
LifeHatesMe wrote:
Why not switch to a more reliable VPS provider like linode? Just curious.


Because Liquidweb is better, incredibly less expensive and faster. Those are just matters of opinion, but they count for a lot with me.

The whole reason people use Nginx, is that there is little reason to load PHP into memory just to serve an image. When using apache, there is little reason not to use mod_php, so...

In general, I just feel nginx is the better of the two for my purposes. No, wait, that is not it. More like... I just felt like it at the time.

Though to be fair, you were right about something being wrong with DNS.



As you might have noted, the server is now up again, though it still seems to not be processing any uploads (or not saving or showing it anyway). I'm going to investigate that now.

I'm sorry for my inability to really take care of the website, but there are just more important things in my life right now.

UPDATE: The problem that was causing the non-processed uploads was a 30GB task queue, caused by my inadvertently not turning on a certain processing task (the one dealing with the information you see on pretty much any page), every upload caused about 6 or 7 of these to spawn in the task queue, so it quickly became clogged up, and mysql just couldn't deal with the enormous table anymore.

I will also be switching over to a SSD based server sometime soon, hopefully making all the disk access mysql must do an order of magnitude faster.

UPDATE 2: The server is now smart enough to restart processing itself if it by any chance decides to shutdown and restart for any reason. I hope this will protect us against the humongous amounts of processing that had to be done after such occurrences.

UPDATE 3: Hopefully, the 502 warnings are now also gone for the forseeable future.

UPDATE 4: If anyone ever wanted to work on EVE Marketeer but thought it wasn't possible because he couldn't get the data, here you go: http://www.evemarketeer.com/var/dump/

It contains an export of all non sensitive data that is required to make the site work. If you always wanted to work with huge sets of data, this is for you as well :P

Developer/Creator of EVE Marketeer

Vaerah Vahrokha
Vahrokh Consulting
#559 - 2012-04-22 19:17:53 UTC
Today for some reason the upload started working again even on the computer where it would not work.
But I still have one odd thing.

I upload very fresh data, I get your website to take it (I see it in my upload stats and on the client side uploader) but I always get about 2 days older history on your website than the history I sent you. I.e. I sent history for Nocxium up to Apr 20 yet on the website OHLC data dump it only covered up to Apr 18. 4 days old data is quite useless.
Vaerah Vahrokha
Vahrokh Consulting
#560 - 2012-04-22 22:29:58 UTC
Another noobie question. For the first ime Attention I have looked at the market orders. They website says they were updated 1 minute before I read them.

Yet in game I see Nocx @Jita CNAP at 985 or so on sell orders, but I see a number of website sell orders @Jita CNAP on the website for about 531 ISK pu before the 985 ones. What gives?