EVE Technology Lab

what are you trying to archive? Market data caching but what exactly. I ask as the third party apps you have strung together here are not compatible and are not designed for what your after.

If you do this, be prepared to handle quite a lot of data. I used to store the raw compressed data from EMDR and it came to 150 to 200 megabytes an hour if I remember right. Also duplicates are quite common, especially in History data, so you need to filter that out before entering the new data into your database. Good luck!

Thanks for all the suggestions. I started setting up a connection to EMDR, but haven't complete it. I also was concerned the data might be considerable. I also was wondering how long it would be before I saw the data I submit to it (via Evemon).

I used to use EveOrg App (https://forums.eveonline.com/default.aspx?g=posts&t=28455) but it looks to be dead.
I run Javascript on the IGB and the EveOrg App would scrape the files, inserting the market data into a local MySQL DB.

Yes I would like history.

I see evemon has a built in scraper, and allows me to add an endpoint (http), which i figured I could run some PHP to grab the data and somehow get it into MySQL DB.

Or I could see how to get EVE Market Data Uploader (EMDU) to do the same.
(or Reverence)

I just figured someone had the same need (local cache files to a DB), and had set something up similar.

Any more suggestions are greatly appreciated.

Yes, store each unique order separately. Update quantities & price if they change on a later scrape.

Run SQL to delete old orders once they are past expiry, or delete them if they do not show up in a later cache file.

I'll also want to keep item the daily item history summary (to determine what items have sold for in the past) for all locations.

I don't think I have the skills to do this upon data loading into the DB, but could probably run SQL to del old/unwanted data post import.

Wow very nice, thank you. I'll go set it up and report back - thanks for the code, you went beyond what I was hoping for!

The data runs to around 350kb/s under normal conditions. That's compressed. (I run one of the relays. My outbound is higher as I have a bunch of subscribers to it.).

Exactly what you do with it is up to you. For my personal consumption, I throw away any data with a generated date older than the data that I have stored, which takes care of most of it.

you might want to replace the < ? with a < ?php

It works, but if short tags aren't on, < ? isn't enough.


If that doesn't work, what you can try is:

replace the

$parsed=json_decode($_POST['data'],true);

with the contents of:
http://pastebin.com/jVzx3HPq



Then if you browse to the page, it /should/ insert.

(oops. I meant to update the evesupport reference to a variable, to specify which database to stick it in)

umm.

Well, orderhistory only populates if you look at the item history. if you just look at the order details, you don't get it. so if you're running a market scan, do it twice, one with the orders, one with the history.

I'm not entirely sure why it thinks it's failing. It was loading fine for me.

For it to be seen as a success, it /has/ to return a 1. I don't think it cares about spaces, but it might. If it returns any other text, it will fail. If it doesn't return a 1, it will fail.

I believe so, yes.

Easier to do the filtering for that client side.

Simple to test for, anyway

Anything showing up in your apache error logs?

In the end, that's how you debug this kind of thing.

That, and adding error_log() statements to checkpoint the code.