These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE General Discussion

 
  • Topic is locked indefinitely.
 

Is there any way to export all the market data without having to click in every item?
Author
Previous Topic Next Topic
QFX
Native Freshfood
Minmatar Republic
#1 - 2011-12-31 05:23:27 UTC
Is there any way to export all the market data without having to click in every item?


Hi Im not looking for any special program or whatever secret some of you might know and not want to share ^^ I just like to export all the market data from one region which I am located at, without having to go through every single item and click export.


Is there anyway to do this?

Much thanks in advance.
Petrus Blackshell
Rifterlings
#2 - 2011-12-31 05:55:09 UTC  |  Edited by: Petrus Blackshell
Try Eve-Central. They have a market data collection tool that lots of people use, and they provide giant daily market dumps, and small API queries for on-the-fly use.

I use their service in multiple ventures of mine, and it's great. The only way in which it's not very dependable is in low-volume items.

Edit: but to answer your question, no, there is no way. That would encourage automation way too much, and make it way too easy.

Accidentally The Whole Frigate - For-newbies blog (currently on pause)
QFX
Native Freshfood
Minmatar Republic
#3 - 2011-12-31 06:35:58 UTC
Petrus Blackshell wrote:
Try Eve-Central. They have a market data collection tool that lots of people use, and they provide giant daily market dumps, and small API queries for on-the-fly use.

I use their service in multiple ventures of mine, and it's great. The only way in which it's not very dependable is in low-volume items.

Edit: but to answer your question, no, there is no way. That would encourage automation way too much, and make it way too easy.


You're wrong.

There is a way, I just need someone to tell me the download link of it.

Thanks tho.
rodyas
Tie Fighters Inc
#4 - 2011-12-31 06:51:21 UTC
Should ask this in the market forum, that is if they feel like sharing their secrets with you.

Signature removed for inappropriate language - CCP Eterne
c4 t
Cosmic Psychedelics
#5 - 2011-12-31 08:43:58 UTC
QFX wrote:
Petrus Blackshell wrote:
Try Eve-Central. They have a market data collection tool that lots of people use, and they provide giant daily market dumps, and small API queries for on-the-fly use.

I use their service in multiple ventures of mine, and it's great. The only way in which it's not very dependable is in low-volume items.

Edit: but to answer your question, no, there is no way. That would encourage automation way too much, and make it way too easy.


You're wrong.

There is a way, I just need someone to tell me the download link of it.

Thanks tho.


maybe try this program i picked up called Sporkeh
Zelda Wei
New Horizon Trade Exchange
#6 - 2011-12-31 09:33:33 UTC  |  Edited by: Zelda Wei
QFX wrote:
Petrus Blackshell wrote:
Try Eve-Central. They have a market data collection tool that lots of people use, and they provide giant daily market dumps, and small API queries for on-the-fly use.

I use their service in multiple ventures of mine, and it's great. The only way in which it's not very dependable is in low-volume items.

Edit: but to answer your question, no, there is no way. That would encourage automation way too much, and make it way too easy.


You're wrong.

There is a way, I just need someone to tell me the download link of it.

Thanks tho.


There is no simple way to download the current entire regional market, note the words current and entire.

Eve-central provides the most complete coverage available but it is still incomplete especially anywhere that is not Jita. The more remote a region is the worse the coverage and the older the data.

There are cache scrapers like libevecache but these will only provide items you've already viewed normally.

There are a few IGB scripts that will freshen the cache by browsing the market for you, Eve Mentat includes one but is otherwise very kludge.
Petrus Blackshell
Rifterlings
#7 - 2011-12-31 10:34:31 UTC
QFX wrote:
Petrus Blackshell wrote:
Try Eve-Central. They have a market data collection tool that lots of people use, and they provide giant daily market dumps, and small API queries for on-the-fly use.

I use their service in multiple ventures of mine, and it's great. The only way in which it's not very dependable is in low-volume items.

Edit: but to answer your question, no, there is no way. That would encourage automation way too much, and make it way too easy.


You're wrong.

There is a way, I just need someone to tell me the download link of it.

Thanks tho.


Oh, I'm sorry, I didn't realize GD was now the place to ask how to bot. No, there is no way to get the whole current market without a lot of clicking. You can make recording the data easier with some of the tools people mentioned, but you still have to do all the clicking.

Accidentally The Whole Frigate - For-newbies blog (currently on pause)
QFX
Native Freshfood
Minmatar Republic
#8 - 2011-12-31 11:39:53 UTC  |  Edited by: QFX
Zelda Wei wrote:
QFX wrote:
Petrus Blackshell wrote:
Try Eve-Central. They have a market data collection tool that lots of people use, and they provide giant daily market dumps, and small API queries for on-the-fly use.

I use their service in multiple ventures of mine, and it's great. The only way in which it's not very dependable is in low-volume items.

Edit: but to answer your question, no, there is no way. That would encourage automation way too much, and make it way too easy.


You're wrong.

There is a way, I just need someone to tell me the download link of it.

Thanks tho.


There is no simple way to download the current entire regional market, note the words current and entire.

Eve-central provides the most complete coverage available but it is still incomplete especially anywhere that is not Jita. The more remote a region is the worse the coverage and the older the data.

There are cache scrapers like libevecache but these will only provide items you've already viewed normally.

There are a few IGB scripts that will freshen the cache by browsing the market for you, Eve Mentat includes one but is otherwise very kludge.


Thanks, I use eve mentat already, should be easy to get something going.



Petrus Blackshell wrote:


Oh, I'm sorry, I didn't realize GD was now the place to ask how to bot. No, there is no way to get the whole current market without a lot of clicking. You can make recording the data easier with some of the tools people mentioned, but you still have to do all the clicking.


Obviously you dont know whats been talking about on here, stop trying to look like you know anything.
malaire
#9 - 2011-12-31 12:30:59 UTC  |  Edited by: malaire
Petrus Blackshell wrote:
Oh, I'm sorry, I didn't realize GD was now the place to ask how to bot. No, there is no way to get the whole current market without a lot of clicking. You can make recording the data easier with some of the tools people mentioned, but you still have to do all the clicking.

You can use Javascript program in IGB to avoid that clicking. You can even minimize EVE while that program is running and do something else on your computer while information is being collected.

You just need typeIDs for all items you are interested, and then javascript-loop which will call CCPEVE.showMarketDetails(typeID) once for each typeID and then wait* a bit between calls. After that all the information is in EVE cache and then you just use cache-reading to get the information.

This kind of automation is fully accepted by CCP (I even petitioned it to make sure it is allowed.)

Small disclaimer: If you really want information about all the items, that will take hours, and once it is finished information for first items isn't current anymore.

EDIT: *) I am not sure how large wait is needed. There seems to be two limitations: Firstly you must not call that function more than once per second. If you only have few items you can get away with e.g. 1.5s wait. But if you have a lot of items in list, eventually that call stops working with small wait since there seems to be something like "you are allowed to use max N calls in X minutes". Personally I use 10 second wait when not in hurry just to be sure.

New to EVE? Don't forget to read: The Manual * The Wiki * The Career Options * and everything else
QFX
Native Freshfood
Minmatar Republic
#10 - 2011-12-31 13:07:38 UTC
malaire wrote:
Petrus Blackshell wrote:
Oh, I'm sorry, I didn't realize GD was now the place to ask how to bot. No, there is no way to get the whole current market without a lot of clicking. You can make recording the data easier with some of the tools people mentioned, but you still have to do all the clicking.

You can use Javascript program in IGB to avoid that clicking. You can even minimize EVE while that program is running and do something else on your computer while information is being collected.

You just need typeIDs for all items you are interested, and then javascript-loop which will call CCPEVE.showMarketDetails(typeID) once for each typeID and then wait* a bit between calls. After that all the information is in EVE cache and then you just use cache-reading to get the information.

This kind of automation is fully accepted by CCP (I even petitioned it to make sure it is allowed.)

Small disclaimer: If you really want information about all the items, that will take hours, and once it is finished information for first items isn't current anymore.

EDIT: *) I am not sure how large wait is needed. There seems to be two limitations: Firstly you must not call that function more than once per second. If you only have few items you can get away with e.g. 1.5s wait. But if you have a lot of items in list, eventually that call stops working with small wait since there seems to be something like "you are allowed to use max N calls in X minutes". Personally I use 10 second wait when not in hurry just to be sure.


Thanks, thats more of an answer that I was looking for.

I guess the wait time all depends on your connection and how smooth your pc performs it, I got a script going but can only check the items from my current orders, I still need to find out how to do it the entire market, like "ships, equipmpent, modifications and ammo".
malaire
#11 - 2011-12-31 13:54:52 UTC  |  Edited by: malaire
QFX wrote:
I guess the wait time all depends on your connection and how smooth your pc performs it, I got a script going but can only check the items from my current orders, I still need to find out how to do it the entire market, like "ships, equipmpent, modifications and ammo".

1) download http://zofu.no-ip.de/inca10/inca10-mysql5-sql-v1/inca10-invTypes-mysql5-v1.sql.bz2 and unpack it

You can get itemIDs you want from there, e.g.
...
INSERT INTO `invTypes` VALUES (638,27,'Raven','The Raven is the ...
INSERT INTO `invTypes` VALUES (639,27,'Tempest','The Tempest is ...
INSERT INTO `invTypes` VALUES (640,27,'Scorpion','The first Scor...
INSERT INTO `invTypes` VALUES (641,27,'Megathron','The Megathron...
INSERT INTO `invTypes` VALUES (642,27,'Apocalypse','In days past...
INSERT INTO `invTypes` VALUES (643,27,'Armageddon','The mighty A...
INSERT INTO `invTypes` VALUES (644,27,'Typhoon','Much praised by...
INSERT INTO `invTypes` VALUES (645,27,'Dominix','The Dominix is ...
INSERT INTO `invTypes` VALUES (647,27,'Eidolon','Little is known...
INSERT INTO `invTypes` VALUES (648,28,'Badger','The Badger-class...
INSERT INTO `invTypes` VALUES (649,28,'Badger Mark II','The Badg...
INSERT INTO `invTypes` VALUES (650,28,'Iteron','The Iteron-class...
INSERT INTO `invTypes` VALUES (651,28,'Hoarder','The Hoarder is..
...

Raven = 638, Tempest = 639, Scorpion = 640, Megathron = 641, ....

2) Make html-page with javascript-loop using those IDs. Something like http://pastie.org/private/ppewro9bn30nwif94cjmqq

3) Open that page in IGB, if you saved that in C:\idloop.htm, use "file:///C:///idloop.htm" in IGB.

4) Click "Start Loop" and wait for script to go through all items.

5) Use cache-reading to get information from cache

6) ....

7) PROFIT

New to EVE? Don't forget to read: The Manual * The Wiki * The Career Options * and everything else