These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

RETURNED - EVE Marketeer - Market Data Aggregation & Information
Author
Previous Topic Next Topic
Callean Drevus
Perkone
Caldari State
#21 - 2011-09-25 08:12:07 UTC  |  Edited by: Callean Drevus
I'm on it!

UPDATE: Fixed it has been

Developer/Creator of EVE Marketeer
Shingyoku
Griffin Support Services
Ad-Astra
#22 - 2011-09-25 15:44:04 UTC
Well this is kind of embarrassing, I usually use keepass for my passwords and it seems to have not saved my password when I setup my account. I also noticed there's no password reset on the site, any plans to implement that?
Callean Drevus
Perkone
Caldari State
#23 - 2011-09-25 22:12:01 UTC
Shingyoku wrote:
Well this is kind of embarrassing, I usually use keepass for my passwords and it seems to have not saved my password when I setup my account. I also noticed there's no password reset on the site, any plans to implement that?


Ah, yes, this is originally an improvement on a corporation website I made, and if anyone lost their password there I could simply reset it myself. I'll send you a new password soon, and make a way to easily reset it yourself (which is going to be interesting, as I do not save any email adresses by default).

Developer/Creator of EVE Marketeer
Domi Etymology
Violent Alternatives
#24 - 2011-09-26 01:16:26 UTC
Just send them an ingame eve-mail. No emails required :P
Shingyoku
Griffin Support Services
Ad-Astra
#25 - 2011-09-26 04:24:10 UTC
Domi Etymology wrote:
Just send them an ingame eve-mail. No emails required :P


I really like this idea. Use the trust feature in the browser or something.
Gretzaki
The New Federation
Shadow Ultimatum
#26 - 2011-09-26 14:21:39 UTC
Site is down again, it's hard to use the data that is being collected when you can't access it :(
Callean Drevus
Perkone
Caldari State
#27 - 2011-09-26 16:52:02 UTC
Gretzaki wrote:
Site is down again, it's hard to use the data that is being collected when you can't access it :(


I am very sorry about this. I assure you, I am not doing it on purpose and am trying to track down what is going wrong. Previously, I knew it had something to do with an unlimited number of threads spawning, but now it just seems to decided to die every 1.5 days.

I can't always be online however, so it may take some time to restart if it isn't working (actually, the remote reboot doesn't work either. It just totally loses connection. Only a complete re-initialization of the VM brings it back online).

And... The logs show nothing! :S sorry, but it may take a while, or it may be fixed tomorrow, that's all I can really say about it Sad

Developer/Creator of EVE Marketeer
Shingyoku
Griffin Support Services
Ad-Astra
#28 - 2011-09-26 19:59:30 UTC
Callean Drevus wrote:
Gretzaki wrote:
Site is down again, it's hard to use the data that is being collected when you can't access it :(


I am very sorry about this. I assure you, I am not doing it on purpose and am trying to track down what is going wrong. Previously, I knew it had something to do with an unlimited number of threads spawning, but now it just seems to decided to die every 1.5 days.

I can't always be online however, so it may take some time to restart if it isn't working (actually, the remote reboot doesn't work either. It just totally loses connection. Only a complete re-initialization of the VM brings it back online).

And... The logs show nothing! :S sorry, but it may take a while, or it may be fixed tomorrow, that's all I can really say about it Sad


Welcome to website admin, isn't it lovely? :D
Callean Drevus
Perkone
Caldari State
#29 - 2011-09-27 16:52:53 UTC
Shingyoku wrote:
Domi Etymology wrote:
Just send them an ingame eve-mail. No emails required :P


I really like this idea. Use the trust feature in the browser or something.


I would love to, but sending Ingame mail cannot be automated, so if you are okey with waiting for me to send a mail, thats fine :P

Developer/Creator of EVE Marketeer
Shingyoku
Griffin Support Services
Ad-Astra
#30 - 2011-09-27 17:29:34 UTC
Callean Drevus wrote:
Shingyoku wrote:
Domi Etymology wrote:
Just send them an ingame eve-mail. No emails required :P


I really like this idea. Use the trust feature in the browser or something.


I would love to, but sending Ingame mail cannot be automated, so if you are okey with waiting for me to send a mail, thats fine :P


Oh I'm ok with that for now, just saying for the future reset system.
Turhan Bey
EVE University
Ivy League
#31 - 2011-09-28 02:14:52 UTC  |  Edited by: Turhan Bey
I strongly recommend against using IGB trust to disseminate lost passwords. It is one of the easier things to spoof. Registering an email address on account creation would be the ideal "easy" solution.

EDIT: However, sending in-game EVEmails for password reminder/reset could be secure if the account has a CAK registered with character info access. But as stated above, it cannot be automated.
Callean Drevus
Perkone
Caldari State
#32 - 2011-09-28 09:46:00 UTC  |  Edited by: Callean Drevus
Turhan Bey wrote:
I strongly recommend against using IGB trust to disseminate lost passwords. It is one of the easier things to spoof. Registering an email address on account creation would be the ideal "easy" solution.

EDIT: However, sending in-game EVEmails for password reminder/reset could be secure if the account has a CAK registered with character info access. But as stated above, it cannot be automated.

I am not planning on using the IGB for any authentication whatsoever :)

It might just be safer and best to do those few times it happens manually. Until CCP builds in automatic sending of IG mails (spam ftw?).

Developer/Creator of EVE Marketeer
Scrapyard Bob
EVE University
Ivy League
#33 - 2011-09-29 01:51:12 UTC
Quote:
LOGSQL Insert Failed: insert into adodb_logsql (created,sql0,sql1,params,tracer,timer) values( NOW(),?,?,?,?,?)
Table './admin_icras/adodb_logsql' is marked as crashed and last (automatic?) repair failed


http://www.evemarketeer.com/statistics/uploads/24

This is showing up at the top of a few pages as well as in the upload tool.
Dragonaire
Here there be Dragons
#34 - 2011-09-29 03:47:16 UTC
Looks like the DB isn't keeping up and crashing for some reason I'd make sure it's not filling up it's log space. I've ran into default configs where every few days that was happening to me as well. There's some settings in my.cnf you should check and make sure it's not configured to keep them forever only like last 24 hours or so. Having thre extra query logging on also add a lot of extra load so except if you just turned it on to figure out what's going wrong I'd have it off on a production server.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Turhan Bey
EVE University
Ivy League
#35 - 2011-09-29 17:54:49 UTC
I've noticed that when receiving these errors, it will sometimes work without fault, but then it will generate errors continuously for an even longer time. The ratio varies greatly, but seems to average perhaps 10 errors to 1 success.

I've also seen the Upload Backlog with greatly skewed values against EVE-MarketData. For example, as I write this, I see the backlog as:

EVE Marketeer: 2
EVE Marketdata: 127
EVE Central: 0

Just a guess, but perhaps it is the update rate from the EMD feed that is throttling the database? Can it be slowed down to see if doing so has an effect on the error rate?
Callean Drevus
Perkone
Caldari State
#36 - 2011-09-29 18:15:11 UTC  |  Edited by: Callean Drevus
Hey, thank you for your replies and help trying to get it fixed.

I made a post a few hours ago, but it seems it didn't get trough. First of all, I've installed logging software, that at least gives me an idea of what happens. If you feel interested, you can view the server stats via Munin

As you can see, there isn't anything shocking there, nothing suddenly happens, except a sudden spike in the usage of one disk. I've yet to find out what is causing this.

To clear two things up. It is not the error you've been seeing about logging, at least it seems very unlikely. That is just due to me neglecting to recover that table after the last crash (it's pretty much always writing to the log table, so every crash kills it :P).

It does not seem to be the database either, as I've been monitoring that, and it steadily keeps at about 5-10 connections doing queries a second, which isn't enough to kill it.

The same goes for the EMD backlog. EMD just gets 130+ updates per minute and I only process them only once per minute, so it might happen that 130 are waiting.

Update: It might be google, trying to index the website :P

Update II: The logs seem to be fine. No huge files there.

Developer/Creator of EVE Marketeer
Gretzaki
The New Federation
Shadow Ultimatum
#37 - 2011-09-30 03:35:18 UTC  |  Edited by: Gretzaki
Callean Drevus wrote:
Hey, thank you for your replies and help trying to get it fixed.

Update: It might be google, trying to index the website :P

Update II: The logs seem to be fine. No huge files there.



If you really think it might be a google index, try using a robots txt to ignore that directory of the site.

Also I would be interested in helping with development in any way I can. Please let me know.

and btw the backlogs are piling up....

EVE Marketeer 6355
EVE Marketdata 25
EVE Central 0
Domi Etymology
Violent Alternatives
#38 - 2011-09-30 06:53:50 UTC
my uploader keeps giving me this:

Error: Upload unknown problem:
Warning: mysql_connect(): Lost connection to MySQL server at 'reading initial communication packet', system error: 111 in /var/www/core/lib/adodb5/drivers/adodb-mysql.inc.php on line 362
1

?
Vessper
Dark Mason Society
#39 - 2011-09-30 14:40:23 UTC
Region XML download is giving errors again:

Warning: mysql_connect(): Lost connection to MySQL server at 'reading initial communication packet', system error: 111 in /var/www/core/lib/adodb5/drivers/adodb-mysql.inc.php on line 362
Callean Drevus
Perkone
Caldari State
#40 - 2011-09-30 20:13:17 UTC  |  Edited by: Callean Drevus
Ok, I basically removed every server package that had anything to do with serving up pages and reinstalled the latest versions, and sometimes even different packages (I'm now using nginx instead of apache, and running php as cgi instead of module). In any case, I hope it solves some of the problems.

Also, I might need to start processing EMD orders in the same way I now process EMK orders, since that seems to be the fastest way, and I now get an immense backlog of EMD orders :P or I could just work on making order processing faster.

Gretzaki, expect an ingame mail about helping development soon.

Developer/Creator of EVE Marketeer