These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

Market Discussions

 
  • Topic is locked indefinitely.
 

Pulling data with Excel

Author
K'rashu Graylock
#1 - 2012-02-07 16:40:37 UTC
I'm a spreadsheet nublet, so I'm wondering if there's a way to yank data from a non-table from the internets.

Further, how do you (proverbial you) pull market data into Excel? I don't need a tutorial inasmuch as I need a workflow/pipeline.

Jesus_Christ renamed to Scoreboard. Scoreboard kills himself.

Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#2 - 2012-02-07 17:20:44 UTC
Go take a look at the eve market data sites (eve central, eve marketdata, eve marketeer). I'd suggest running the unified uploader from eve marketeer, to keep the data fresher (it can update all three)

They provide apis to retrieve data from various regions, which you can then use in Excel.

If you want to use your own data, that's substantially more complex. Either you'd use reverence and have some kind of parsing going on, or you'd run a local webserver, with a endpoint script, with the unified uploader pointing at it, and loading the data into a local database (where excel can than get at it with odbc) If you want to try that route, there's a sample uploader in the resources sidebar of my site. You could try using it with XAMPP providing the mysql, apache and php.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

Scrapyard Bob
EVE University
Ivy League
#3 - 2012-02-07 18:39:16 UTC
The problem with Excel and LibreOffice Calc is that it is not intuitive or straightforward to pull XML data sources in and work with them. You have to jump through a lot of hoops to do so.

Google Docs, OTOH, makes this dead simple to do with the ImportXML() call.

Now, there is a way to do an equivalent to the ImportXML() call in Excel or LibreOffice, but every time I look into doing it, I realize it looks like work and just go back to using Google Doc's ImportXML() call.

To pull XML sources, you need to know:

1) the TypeIDs of the items that you are looking for (see Fuzzwork's site, or Chribba's TypeID listing, or look through the invtypes table in the CCP static data dump)

2) The region ID, if you want something other then the default

3) How to put #1 and #2 together to form the XML URL to retrieve the data that you want

4) How to construct the XPath filter argument to the ImportXML() call to get what you want (unless you want a big 2D list of numbers)

Which gives you the ability to make calls like:

https://docs.google.com/spreadsheet/ccc?key=0AsopBfkYYvf3cDN5LXVZTmwtM0NYZWZCZVZIWVBpSEE&hl=en_US#gid=4

See the Prices1 sheet, where I demonstrate how to pull prices for the various minerals from EveMarketeer, Eve-MarketData and Eve-Central. It helps to use Chrome or Firefox to view the resulting XML URL to see the actual structure of the returned XML data.

EMD's version:
http://api.eve-marketdata.com/api/importxml_prices2.xml?char_name=EUNI&buysell=s&region_id=10000030&type_ids=34,35,36,37,38,39,40,11399,16272,16273,16275,16274,17888,17887,17889,4247,4051,4312,4246

EMK's version:
http://www.evemarketeer.com/api/info/34_35_36_37_38_39_40_11399_16272_16273_16275_16274_17888_17887_17889_4247_4051_4312_4246/xml/10000030/sell_lowest5

EC's version:
http://api.eve-central.com/api/marketstat?typeid=34&typeid=35&typeid=36&typeid=37&typeid=38&typeid=39&typeid=40&typeid=11399&typeid=16272&typeid=16273&typeid=16275&typeid=16274&typeid=17888&typeid=17887&typeid=17889&typeid=4247&typeid=4051&typeid=4312&typeid=4246&regionlimit=10000030
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#4 - 2012-02-07 19:07:10 UTC
Really? I found Excel (at least 2010) dead simple to use with xml. Just use an 'import from web', hand it the xml feed, and it'll ask how you want to use it. giving a full xml table if you want it to.

Open/Libre office on the other hand is, indeed, a fair bit more difficult to use.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

Scrapyard Bob
EVE University
Ivy League
#5 - 2012-02-08 00:12:40 UTC
Ah, I have not tried out my copy of Excel 2010 yet. I shall have to look at that for my more ambitious sheets.
Nnam Pir
Nnam Fleet
#6 - 2012-02-11 07:14:11 UTC
Has anyone done this sort of thing in Open Office Calc?
Bussa Braun
Pator Tech School
Minmatar Republic
#7 - 2012-02-12 17:48:12 UTC
There used to be a spot under your character information where you could pull your "market transactions" and your "journal history", but the b@st@rds moved or deleted it from the new website and they wont tell anybody where it is X!

You can still export your market buy and sell orders straight to excel from the market orders tab in your wallet.

MrVilla
Caldari Provisions
Caldari State
#8 - 2012-02-27 09:42:23 UTC
Hi

When i export my orders i do not get the item name, only the type ID.
Anyone have a table with typeID and names so I can do a lookup function and see what it all is?:)
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#9 - 2012-02-27 12:18:59 UTC
MrVilla wrote:
Hi

When i export my orders i do not get the item name, only the type ID.
Anyone have a table with typeID and names so I can do a lookup function and see what it all is?:)



There's the table in the static data dump. There are many different formats available. just search in the tech forum.

If all you're wanting is a csv you can import and use with vlookup, grab the csv from my site.

It's
typeid,typename,typeid
So you can use the same sheet for vlookups with the id and vlookups with the name

If you're using openoffice/libreoffice, you can link it directly, with an Insert->sheet from file, giving it the full url. and if you tick the box, it'll stay up to date (I'll be updating it each time the underlying data changes. Like with the missile name changes)

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

TheSmokingHertog
Julia's Interstellar Trade Emperium
#10 - 2012-02-27 18:22:27 UTC
Steve Ronuken wrote:
MrVilla wrote:
Hi

When i export my orders i do not get the item name, only the type ID.
Anyone have a table with typeID and names so I can do a lookup function and see what it all is?:)



There's the table in the static data dump. There are many different formats available. just search in the tech forum.

If all you're wanting is a csv you can import and use with vlookup, grab the csv from my site.

It's
typeid,typename,typeid
So you can use the same sheet for vlookups with the id and vlookups with the name

If you're using openoffice/libreoffice, you can link it directly, with an Insert->sheet from file, giving it the full url. and if you tick the box, it'll stay up to date (I'll be updating it each time the underlying data changes. Like with the missile name changes)


Thx for your CSV, its very handy to have.

"Dogma is kind of like quantum physics, observing the dogma state will change it." ~ CCP Prism X

"Schrödinger's Missile. I dig it." ~ Makari Aeron

-= "Brain in a Box on Singularity" - April 2015 =-