These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Any Market Data

First post
Author
Previous Topic Next Topic
Barrak
The Painted Ones
#1 - 2015-06-13 08:47:26 UTC
Hi,

Hopefully this is a very simply question for you to answer from someone that is slowly (incredibly slowly) getting into coding etc.

Does the introduction of the Crest programs allow you to pull real time data from the game?

Previously things like EvE-Central were great for Jita but not elsewhere as they were not updated well enough. Does Crest resolve this or is it still reliant on people going into the game and uploading stuff?

Hopefully my questions make sense but they are from someone that really understands nothing about this and has only just started to learn.

Regards

Barrak
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#2 - 2015-06-13 16:12:45 UTC
Almost real time.

The first person to request gets it live. Then that's the cached answer for the next 5 minutes, which everyone gets.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Barrak
The Painted Ones
#3 - 2015-06-13 19:46:40 UTC  |  Edited by: Barrak
Steve Ronuken wrote:
Almost real time.

The first person to request gets it live. Then that's the cached answer for the next 5 minutes, which everyone gets.


Hi Steve,

Thanks for the reply. As you know I'm just doing a lot of research at the moment and trying to get things to fall into place.

I managed to build a simple googledocs page that grabs Jita buys/sells that I'm delighted with (yup .... very new to this).

The above gets it's information through EvE-Central. Is that live or thereabouts?

I'm not entirely sure I'm asking the right question or in the right way.

My understanding previously was that a pilot had to download some data from his visit to a station to ensure that EvE-Central was up-to-date.

Based on this:


  1. Is that still the case?
  2. Does CREST allow information to be drawn from EvE without pilots updating it?


I have a few questions in relation to my googledocs too if I might ask them here.


  1. Is there a limit to the number of TypeID's I can query? (I'm using a 'join' command to get 17 ID's at the moment)
  2. Are the items/ID's listed in a catagory somewhere? For instance if I wanted to look at all frigates or HAM's, can i do that through ID's (or anything else) or do I have to manually update that.


Regards

BArrak

ps. Just as a last thing. I noticed my google docs just lost all if its pulled data. Any reason for that and how to I make it call it again?
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#4 - 2015-06-13 20:07:29 UTC
Eve Central, iirc, now pulls data from CREST. It's not live, but it's better than it used to be. (It takes time to get full regions worth of information.)

CREST allows one person (or site) to pull the entire market, one item at a time. It can be fully automated.

For pulling from Eve Central, there aren't many limits on what you can pull. (my function pulls around 150 a time, but is designed so you can ask the function for thousands, if you really want. it just breaks it up)

You still have to pull them individually, rather than asking for all of them at once.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Barrak
The Painted Ones
#5 - 2015-06-13 20:39:33 UTC  |  Edited by: Barrak
Thanks.

Is it possible to group the ID's into, for example, missiles, Hull upgrades etc?

What I want to do is to be able to query data in batches rather than requesting tons of stuff. I also do not want to query things that are not market items.

Thus, what I really need is a file similar to what I downloaded from Fuzzworkds (TypeIDs) but with group ID's attached AND only (or an indicator showing) market items.

Does such a thing exist?

I'm also looking for an explanation as to what many of the column headers are.

I took this from a blog. He stats that Q is his sale price but it doesnt match any item I have looked at after querying the data. Thus I am trying to work out why he uses Q and whilst I'm at it, seeing if i can use anything else.

A version
B method
C id
D volume
E avg
F max
G min
H stddev
I median
J percentile
K volume2
L avg3
M max4
N min5
O stddev6
P median7
Q percentile8
R volume9
S avg10
T max11
U min12
V stddev13
W median14
X percentile15

Regards

Barrak
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#6 - 2015-06-14 03:47:39 UTC
https://www.fuzzwork.co.uk/dump/latest/invTypes.xls.bz2 is the full invTypes table, rather than one cut down to just name and id.

it includes a group id (from https://www.fuzzwork.co.uk/dump/latest/invGroups.xls.bz2 ) which can be linked to a category id (categories are like 'ship' and 'module')

it also has the market group id (which links to https://www.fuzzwork.co.uk/dump/latest/invMarketGroups.xls.bz2)

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Barrak
The Painted Ones
#7 - 2015-06-14 07:16:55 UTC  |  Edited by: Barrak
Steve Ronuken wrote:
https://www.fuzzwork.co.uk/dump/latest/invTypes.xls.bz2 is the full invTypes table, rather than one cut down to just name and id.

it includes a group id (from https://www.fuzzwork.co.uk/dump/latest/invGroups.xls.bz2 ) which can be linked to a category id (categories are like 'ship' and 'module')

it also has the market group id (which links to https://www.fuzzwork.co.uk/dump/latest/invMarketGroups.xls.bz2)


Fantastic Steve. Thank you very much.

*edit*

I can't open the last link for some reason.

I assume that if it does not have a market ID then it can not be purchased/sold on the market?

Obviously with my next comment I'm expecting you to have assimilated all of this information into your head Blink

Taking "Lead Charge L" as an example it has:

TypeID 234
GroupID 85
MarketGROUPID 106

I was going to say that GroupID is 'Ammunition & Charges' but then I noticed that Crystals are 86 and Cap Boosters are 87 thus 85 must signify Hybrid Charges and the 106 is Large.

TypeID 234
GroupID 85 Hybrid Charges
MarketGROUPID 106 Large


  1. Are there ID's for Ammunication & Charges? or the equivalents at this level of ID
  2. Is there a table I can look at that gives me the names as well as the Market/GroupID's?
  3. Based on the work I've done so far, which uses 'vlookup' I can use the invTypes instead of the more basic file which I grabbed a while back (just typeID).



NOTE:

Steve/All - I'm very conscious that I am asking for a lot of information/handouts and I do not want to abuse your time and generosity thus far.

On doing some research I found this older reply which details how to get the informaiton I need but I think its beyond my skills at this stage (most things are sadly). However if it is easier to point me to somewhere I can learn this then I'll happily try rather than just asking for people to give me stuff over and over.
https://forums.eveonline.com/default.aspx?g=posts&t=15255

**EDIT**

Wondering if it's easier just to ask 'how' I get these files myself. What do I need to do.

Regards

Barrak
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#8 - 2015-06-14 17:41:13 UTC
bah. https://www.fuzzwork.co.uk/dump/latest/invMarketGroups.xls.bz2 (it added the ) to it)

If something doesn't have a market group, it doesn't show up on the market. (also, if something doesn't have published set to 1, it doesn't show up in game)

Unfortunately, the data isn't fun to work with, due to being heriarchical. so to find all the groups for ammo, you'd have to go to the top level ammo group, then get all its children, then all their children, then you can find the members.

If all you really want is charges, the easier way to go about it is by categoryID, via the invGroups table. All charges are category 8.

This is the kind of thing that databases excel at, and spreadsheets tend to break down crying, and having to use nested vlookups.

To get, for example, all the typeids of ammo, with a database, it's just

select invTypes.typeid from invTypes join invGroups on invTypes.groupid=invGroups.groupid where categoryid=8;




If you want to generate files yourself, I'd suggest the following:

Get the sqlite converison of the SDE. https://www.fuzzwork.co.uk/dump/sqlite-latest.sqlite.bz2
Unzip it (7zip)
Open it up with an sqlite tool of some kind. sqlite expert (personal edition) is an option here, though it has no export option. just cut and paste. Which may be enough for you.
run an appropriate query, like the one above.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Barrak
The Painted Ones
#9 - 2015-06-14 17:54:31 UTC
THanks Steve.

I'll take a closer look at that this evening. I think I have a lot of downloading to do.

Regards

Barrak

ps... No idea when the CSM votes took place but congrats! Big smile
Barrak
The Painted Ones
#10 - 2015-06-15 14:33:14 UTC
I'm going to ask what is probably very silly question with an obvious answer.

If I learn enough about SQlite and the databases, am I able to build a database that does what the google-docs does which is to compare prices on items in different systems/regions?

I'm slowly working out (very much trial an error, through cut & paste, rather than any skilled approach) how to build a databse of the information that i want, which at this stage is more about understanding the following:

catagoryID
category Name
groupID
groupName
marketGroupID
marketGroupName

The reason for this is that I want to build a report (database or googledocs) that has each market item in it but sorted by a group/category etc so that I can query one group of information at a time rather than calling for the whole market (though, if that is allowed and is not slow then I could just do that).

Infact it's would appear to be very similar to Steve's replica EvE Market, in terms of categories.

Regards

Barrak
Kale Freeman
Garoun Investment Bank
Gallente Federation
#11 - 2015-06-16 11:58:25 UTC
Steve Ronuken wrote:
Almost real time.

The first person to request gets it live. Then that's the cached answer for the next 5 minutes, which everyone gets.


How does one go about finding out what the end point is for this live(ish) market data?
All I can find is a market history endpoint that gives a daily summary for a region/typeId pair.
Is there a list of endpoints somewhere?
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#12 - 2015-06-16 12:07:19 UTC
A database itself won't do any lookups from a third party service. That's something you'd code in another language which happens to use that database. And you'll still want something to sit in front of the database to be the user interface. But you can do much of the data processing in the database itself.

The key concept for what you have above is joins. 


select typeid,typename,invGroups.categoryID,categoryName,invTypes.groupID,groupName,invTypes.marketGroupID,marketGroupName 
from invTypes
join invGroups on invTypes.groupid=invGroups.groupid
join invCategories on invGroups.categoryid=invCategories.categoryid
join invMarketGroups on invTypes.marketgroupid=invMarketGroups.marketgroupid
join invCategories on invGroups.categoryid;


is an example of how you'd join in each table onto invTypes, to get all the information for all the types. you could put in a where clause which would limit it to a specific category, or type, or market group, fairly easily.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Barrak
The Painted Ones
#13 - 2015-06-16 12:19:06 UTC
Steve,

A little off-topic here as it's not really eve related but SQlite.

I can not get a table to work. Even if i copy yours line for line it draws no information. Am I missing something horrifically basic.

If this is to complicated to resolve here please just say so I and I'll contionue searching for training/soloution.

thanks
Mr Mac
Dark Goliath
#14 - 2015-06-18 21:05:41 UTC  |  Edited by: Mr Mac
Barrak wrote:
Steve,

A little off-topic here as it's not really eve related but SQlite.

I can not get a table to work. Even if i copy yours line for line it draws no information. Am I missing something horrifically basic.

If this is to complicated to resolve here please just say so I and I'll contionue searching for training/soloution.

thanks


there some error with last join
just remove last line
and it should work Blink
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#15 - 2015-06-19 11:21:20 UTC
:blush:

umm, I'm not entirely sure why I was trying to join the invCategories table in again. I /suspect/ it was some kind of mixup with pasting it in, as I had a working query (I've been burnt before, so I do test them. The last one in my history works fine)

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#16 - 2015-06-19 11:35:29 UTC
Kale Freeman wrote:
Steve Ronuken wrote:
Almost real time.

The first person to request gets it live. Then that's the cached answer for the next 5 minutes, which everyone gets.


How does one go about finding out what the end point is for this live(ish) market data?
All I can find is a market history endpoint that gives a daily summary for a region/typeId pair.
Is there a list of endpoints somewhere?



Yes and no.

There's the list at the root of the crest server.

From there, you can get to them all (except the history one, because CCP Foxfour is a bad. Blink )

Unfortunately, the market one links through the region (as it's all regional data. it makes sense, but it's not on the root).

https://github.com/fuzzysteve/CREST-Market-Downloader
is a downloader for market data, if you can read python

https://github.com/fuzzysteve/CREST-Market-Viewer is how to do it in javascript (using implicit auth)

https://github.com/fuzzysteve/CrestLibrary is a Crest library, with a demo on how to get market data. In PHP.


Right now, all this information requires you to be logged in.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter