These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
Previous page12
 

Who is using EMDR?
Author
Previous Topic Next Topic
Desmont McCallock
#21 - 2012-12-27 18:27:21 UTC  |  Edited by: Desmont McCallock
Glad you sort it out. Still you owe me those CSPA I payed for EVE mailing you (just kidding, don't dare send any ISK).
Quantix BlackStar
Blackstar Solutions Inc.
#22 - 2012-12-27 22:33:17 UTC  |  Edited by: Quantix BlackStar
Desmont McCallock wrote:
Code still works fine with clrzmq.2.2.5 (https://github.com/zeromq/clrzmq/downloads).

@Quantix BlackStar
The code doesn't use ZeroMQ installer but the .NET 'clr' binding (http://www.zeromq.org/bindings:clr). So don't rush on judgement.


There were some other changes I made in order to get this to work (I'm using the ZMQ nuget distribution)... I'll diff the two and follow up with you privately if they were significant.

Either way, the EveHQ project will be moving to utilize this data (consuming via eve-central), and also pushing data back up.

Cheers,

Q

EveHQ is a multi-faceted character application for EVE Online. Includes pilot monitoring, skill queue planning, ship fitting, industry and more. Get the latest version at http://evehq.net
EveDataRules
World Domination Inc
#23 - 2012-12-28 03:17:20 UTC
Using it: http://www.evedata.org

Much better than the mail push and http pulls. It would be nice if there were some more examples of pushing the data into a SQL server without crippling it though.

I am currently using shared structures to deal with duplicates, nuking the old data if it was reported later than the new data and using a single thread to post updates to MySQL every 120 seconds while taking advantage of MySQL's insert/update multiple rows feature. Loving it.

Anyone have any other suggestions on pushing data to a database?
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#24 - 2012-12-28 10:01:35 UTC
Depends what data you're storing, really.

I'm using memcache as my primary store as the data I'm interested in tends to be thrown around quickly. Takes around an hour to populate the majority of it. Hourly I throw the data down onto disk in csv and json files. I don't care about the history though. Just percentiles for buying and selling.

If I was wanting something more persistent, I'd probably use memcache as a primary store, and have a process in the background that assembles the inserts and batches them every few minutes.

I suspect the other way would be to use one of the other message queues to hold the 'good' data, if you're wanting a FIFO processing of updates.

so:


EMDR Consumer:
zeromq for accepting data from EMDR
memcache for checking the state you have held (to eliminate repeats and old data)
ActiveMQ/RabbitMQ/Qpid for storing the good non-duplicate data

Your own personal consumer:
client for the broker above
the database.
Runs on a slower cycle for batching the changes.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Muscaat
EVE Markets
#25 - 2012-12-29 15:34:10 UTC
I'm working on a new data-collection backend for EVE Markets using EMDR. No ETA yet on when it might go live though Smile
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#26 - 2012-12-29 16:17:35 UTC
EveDataRules wrote:
Using it: http://www.evedata.org

Much better than the mail push and http pulls. It would be nice if there were some more examples of pushing the data into a SQL server without crippling it though.

I am currently using shared structures to deal with duplicates, nuking the old data if it was reported later than the new data and using a single thread to post updates to MySQL every 120 seconds while taking advantage of MySQL's insert/update multiple rows feature. Loving it.

Anyone have any other suggestions on pushing data to a database?



Oh, I was meaning to add:

Make sure you're using innodb, rather than myisam for the table you're updating. It handles the locking better for updates. myisam is very bad for that.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
EveDataRules
World Domination Inc
#27 - 2012-12-29 19:41:28 UTC
I am using InnoDB, it is very nice, the market update and history update are one single query each, not sure if it will still lock per row or do the whole table in that instance. It is very fast though, around 12k inserts a second on old hardware (3ghz P4, 2gb DDR2 Ram, velociraptor).

I have been thinking about rewriting my consumer into C to make it a little faster and handle memory better. Currently it is written in Perl.

I am not sure about putting another application in the middle of this; is ActiveMQ/RabbitMQ/Qpid for creating a store then writing said store to disk for querying later? Maybe using LOADFILE in MySQL with the dump would be faster?

Right now i am just using shared hashes of arrays in Perl with a key of orderID and something a bit more complicated for the history.
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#28 - 2012-12-29 19:59:54 UTC
EveDataRules wrote:
I am using InnoDB, it is very nice, the market update and history update are one single query each, not sure if it will still lock per row or do the whole table in that instance. It is very fast though, around 12k inserts a second on old hardware (3ghz P4, 2gb DDR2 Ram, velociraptor).

I have been thinking about rewriting my consumer into C to make it a little faster and handle memory better. Currently it is written in Perl.

I am not sure about putting another application in the middle of this; is ActiveMQ/RabbitMQ/Qpid for creating a store then writing said store to disk for querying later? Maybe using LOADFILE in MySQL with the dump would be faster?

Right now i am just using shared hashes of arrays in Perl with a key of orderID and something a bit more complicated for the history.



The inbetween queue is to store the list of 'what needs to be updated'. So it can be handled in a batched fashion. Disk is an option though.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Previous page12