These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
12Next page
 

Learning ESI and More

First post
Author
Previous Topic Next Topic
Haile Korhal
Professional Amateurs
#1 - 2016-11-23 02:25:36 UTC
Are there any resources available to helping us learn how to use ESI? For those of us who have never heard of Swagger before, who are only JUST NOW getting started trying to learn XML (the horror), who haven't programmed in 8+ years (or never?), who are inherently bad at self teaching scripting, and who haven't even bothered trying CREST. Is there any hope?

Is ESI just an Eve Flavoured version of some industry standard programming or scripting language? Should we go learn C++ or Java? I hope not. I know I'm not the only person struggling to understand the XML, and I've heard great things about ESI being so much simpler to understand.

Would a sticky be out of the question, containing links to such resources or suggestions? Maybe books on amazon if such things exist? This would be even more important since CREST/XML is going away in 18 months, according to Steve (somewhere else), so we need to be able to convert our spreadsheets to ESI, along with our programs and such.

Or am I completely misunderstanding what ESI is? i.e. can I rip XML out of my Google Sheet / Microsoft Excel and replace it with ESI?

Also yes, this is a different topic than my Citadel Market Information post on purpose. I would like to discuss the feasibility of learning this new fandangled architecture that will soon dominate the 3rd Party Developer life, which I am inexorably being drawn into (even though I swore to myself when I started my ESS Business that I'd avoid like the plague!). Any help or useful advice, links, or gentle flames would be appreciated.

Thank you.

Egregious Spreadsheet Services - For Spreadsheets as a Service to businesses, corporations, and higher, look no further!
Alt Two
Caldari Capital Construction Inc.
#2 - 2016-11-23 04:59:42 UTC
Have you seen https://esi.tech.ccp.is? It describes all of the ESI endpoints and how to use them.
Take Get character's public information as an example.

It tells you that the endpoint path is /characters/{character_id}/
It tells you what the parameters are. In this case character_id which is a required EVE character id.
It tells you the possible HTTP error codes in case something goes wrong.
And you have the "try it out" button where you can experiment with the endpoint and see how the json response looks.

I haven't found a way to get IDs from ESI but you can use the XML API for that. For example to get my ID you would call https://api.eveonline.com/eve/CharacterID.xml.aspx?names=Alt%20Two and parse the XML.

The only really tricky part is how to use the ESI endpoints which require authentication. Have a look at Introduction to SSO for a guide on how to get an access token. This was written for CREST, but the ESI authentication works almost the same way.

What language are you using? Most, if not all, have 3rd party libraries for parsing JSON and XML so you don't have to write everything from scratch. Some even have libraries specifically made for the EVE APIs. Maybe start with one of those to see how things are done.

Someone else can probably give a much better description than me though. I have only been playing around with ESI for a few days and I had never heard of swagger before this so there are likely lots of stuff I should know about it which I don't (yet) :)
Messenger Of Truth
Butlerian Crusade
#3 - 2016-11-23 18:37:36 UTC
What you need is some sample code. I've knocked this up in python to demonstrate how you might use ESI to find some information about a character.

First it searches on the name of a character to find its ID, then if it finds the ID it searches for the public info of the character, and prints out its gender and bio.

You would need python installed, along with the bravado library (run "pip install bravado") in order for this example to work:

#!/usr/bin/env python
# -*- coding: UTF-8 -*-                                                                        
from __future__ import print_function                                                          
                                                                                               
from bravado.client import SwaggerClient
import bravado.exception                                                                       
                                                                                               
def main():
    
  client = SwaggerClient.from_url('https://esi.tech.ccp.is/latest/swagger.json')               
    
  characterName = "Messenger Of Truth"                                                         
                                                                                               
  charResults = client.Search.get_search(                                                      
            search=characterName, 
            categories=['character'],
            strict=True,                                                                       
            ).result()['character']                                                            

  if len(charResults) <= 0: raise Exception("Character not found")                             
  
  characterId = charResults[0] #assuming only one result
  charInfo = client.Character.get_characters_character_id(character_id=characterId).result()   
  
  print("Name: %s" % (charInfo['name']))                                                       
  print("Gender: %s" % (charInfo['gender']))                                                   
  print("Bio: %s" % (charInfo['description']))                                                 
  
                                                                                               
if __name__ == "__main__":                                                                     
  main()

Trade Hub Price Checker: stop.hammerti.me.uk/pricecheck

Visit "Haulers Channel" in game for all matters courier-related.

Structure name/system API: stop.hammerti.me.uk/api
Zifrian
Federal Defense Union
Gallente Federation
#4 - 2016-11-23 18:47:59 UTC
I think the OP is looking to do things in spreadsheets, that they will no longer be able to do. Right now you can do xml calls and get data pretty easily. I guess the only option would be someone to provide an API that gives us data from ESI?

My biggest issue with ESI is that I need to do auth from my desktop application, which I'm not sure how to do yet. When I first saw oauth, I thought "oh that's for web sites" but now I can't ignore it.

I think it'd be great if someone had a .Net desktop app example to share.

Maximze your Industry Potential! - Download EVE Isk per Hour!

Import CCP's SDE - EVE SDE Database Builder
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#5 - 2016-11-23 20:35:55 UTC
Zifrian wrote:
I think the OP is looking to do things in spreadsheets, that they will no longer be able to do. Right now you can do xml calls and get data pretty easily. I guess the only option would be someone to provide an API that gives us data from ESI?

My biggest issue with ESI is that I need to do auth from my desktop application, which I'm not sure how to do yet. When I first saw oauth, I thought "oh that's for web sites" but now I can't ignore it.

I think it'd be great if someone had a .Net desktop app example to share.



There's a couple of ways to do it.

For you, _probably_ the easiest way would be to register a custom url scheme (like myapp://local/auth ) and have the callback go to that. Then, when they auth, you throw them to their system browser, they auth there, and it redirects them, firing up your application (which checks to see if another copy is running, and if it is, passes the details over in a socket connection) This is also how most mobile apps work. https://msdn.microsoft.com/library/aa767914(v=vs.85).aspx


The other way involved embedding a webserver, and then you use http://localhost:some port/auth to do it. Generally pretty simple and doesn't require a protocol registration, but could have trouble if someone else uses that port. https://blog.appharbor.com/2012/06/21/oauth-for-net-desktop-applications


Once they've authed once, you have a refresh token you can use, in future, to get your access token, removing the requirement to auth.




The _worst_ way to do it, is to embed a web client. If I saw an app doing that, from anyone other than the vendor (CCP in this case), I'd not use it. Because I can't trust the developer not to be stealing credentials.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Haile Korhal
Professional Amateurs
#6 - 2016-11-23 23:50:36 UTC
I'm just looking for some resources on how to get started. First and foremost I run my own little spreadsheet business called Egregious Spreadsheets (link in bio) and I've tried staying away from importing eve data from ccp but too many people keep asking for it, so I started trying to learn. Then CCP said no more XML soonTM, so now I'm wondering if I can do the same things with ESI. If CCP is going to be getting rid of XML without any reliable replacement, how are all of us non-programmer plebs going to manage? I've got dozens of spreadsheets I use on a daily basis to manage everything from membership rosters, asset tracking, reaction planning, PI, pvp, budgeting, etc. If I would have to program my own application, executable and all, for these thing, I'd be pretty screwed! Please tell me they're not going to leave the hundreds or thousands of spreadsheet users in the dust!

In the essence of self preservation, I want to learn how to use ESI, XML, CREST, etc. At the moment my program of choice is Excel and Google Sheets. I would be thrilled if I could learn how to program (as I know the basics with C++ 6.0 about 8 years ago) to do all this stuff better. I will admit automatically importing data is a whole lot nicer than adding data manually cell by cell.

So, install python, is it an actual development environment? I was under the impression that it was just a programming language you use in something like Visual Studio.NET. Should I look into getting something like that again?

There's a lot of information in this thread already so I'm going to spend some time this weekend sorting through it all and see if I can answer some of my questions. Thank you for the input! If anyone else has any input on learning ESI, how to use it, where to go to learn more about it, etc. Please feel free to contribute. I'm not the only one out there who knows little to nothing and wants to learn!

Egregious Spreadsheet Services - For Spreadsheets as a Service to businesses, corporations, and higher, look no further!
Vex Munda
Anti Enslavement Movement
#7 - 2016-11-24 12:22:03 UTC
Haile Korhal wrote:
I'm just looking for some resources on how to get started. First and foremost I run my own little spreadsheet business called Egregious Spreadsheets (link in bio) and I've tried staying away from importing eve data from ccp but too many people keep asking for it, so I started trying to learn. Then CCP said no more XML soonTM, so now I'm wondering if I can do the same things with ESI. If CCP is going to be getting rid of XML without any reliable replacement, how are all of us non-programmer plebs going to manage? I've got dozens of spreadsheets I use on a daily basis to manage everything from membership rosters, asset tracking, reaction planning, PI, pvp, budgeting, etc. If I would have to program my own application, executable and all, for these thing, I'd be pretty screwed! Please tell me they're not going to leave the hundreds or thousands of spreadsheet users in the dust!

In the essence of self preservation, I want to learn how to use ESI, XML, CREST, etc. At the moment my program of choice is Excel and Google Sheets. I would be thrilled if I could learn how to program (as I know the basics with C++ 6.0 about 8 years ago) to do all this stuff better. I will admit automatically importing data is a whole lot nicer than adding data manually cell by cell.

So, install python, is it an actual development environment? I was under the impression that it was just a programming language you use in something like Visual Studio.NET. Should I look into getting something like that again?

There's a lot of information in this thread already so I'm going to spend some time this weekend sorting through it all and see if I can answer some of my questions. Thank you for the input! If anyone else has any input on learning ESI, how to use it, where to go to learn more about it, etc. Please feel free to contribute. I'm not the only one out there who knows little to nothing and wants to learn!


From what I understand xml will be going away completely in the end. ESI will return the info in JSON format (because it takes less file size to send).

If you want to use ESI you are going to have to figure out how to do the SSO (single sign on) to get an authorization key. You need some knowledge about http requests for this. If you don't know already, it's going to take a couple of days to figure out. (tbh I still didn't figure it out completely for my website). Once you are able to get the authorization key, you can start making requests. But figuring the SSO out is definitely a big pain if you're an inexperienced coder.

I don't know if it is possible to use the regular API keys. Which enables you to skip the SSO process. Probably without problems. Maybe someone else can tell you more about this.
Messenger Of Truth
Butlerian Crusade
#8 - 2016-11-24 13:27:55 UTC
Haile Korhal wrote:
So, install python, is it an actual development environment? I was under the impression that it was just a programming language you use in something like Visual Studio.NET.


Python is a programming language, one of its strengths is that there are a lot of libraries available for you to use, but its not the only programming language that can interface with ESI. If you go down the programming language route rather than the spreadsheet route there are many languages that you could use to interface with CCP's APIs - Python, Java, C++, PHP, and many others.

Or, someone will eventually figure out a way to combine SSO with spreadsheets and then you would be able to import data directly into your spreadheets - maybe its even been done already by something like https://github.com/nuadi/googlecrestscript

Trade Hub Price Checker: stop.hammerti.me.uk/pricecheck

Visit "Haulers Channel" in game for all matters courier-related.

Structure name/system API: stop.hammerti.me.uk/api
Haile Korhal
Professional Amateurs
#9 - 2016-11-24 21:49:42 UTC
Maybe it'll even be me! Goodness, well, that sucks a little. I hope someone can figure it out, because spreadsheets are my life! I'd prefer to stick with them and only them, but I realize the world needs other stuff. I used to program a long time ago, so maybe I can pick it back up again. I'm sure the theory hasn't changed all that much and I am familiar with reading code. HTTP is a bit of a struggle but I've heard that from a lot of people.

I'll have to figure out a lot of things probably, sigh. It's time to learn stuff! ESI is the future and if we don't adapt we die, so that's why I started this thread. Thank you everyone for your suggestions so far, if anyone else has any more please feel free to post them. Books, websites, and more, etc.

So what programming environment do you use, or would you recommend?

Egregious Spreadsheet Services - For Spreadsheets as a Service to businesses, corporations, and higher, look no further!
Zifrian
Federal Defense Union
Gallente Federation
#10 - 2016-11-24 21:56:31 UTC
Haile Korhal wrote:
Maybe it'll even be me! Goodness, well, that sucks a little. I hope someone can figure it out, because spreadsheets are my life! I'd prefer to stick with them and only them, but I realize the world needs other stuff. I used to program a long time ago, so maybe I can pick it back up again. I'm sure the theory hasn't changed all that much and I am familiar with reading code. HTTP is a bit of a struggle but I've heard that from a lot of people.

I'll have to figure out a lot of things probably, sigh. It's time to learn stuff! ESI is the future and if we don't adapt we die, so that's why I started this thread. Thank you everyone for your suggestions so far, if anyone else has any more please feel free to post them. Books, websites, and more, etc.

So what programming environment do you use, or would you recommend?

What about using VBA?

Maximze your Industry Potential! - Download EVE Isk per Hour!

Import CCP's SDE - EVE SDE Database Builder
foxjazz
Center for Advanced Studies
Gallente Federation
#11 - 2016-11-28 19:13:38 UTC
Haile Korhal wrote:

What about using VBA?


I think that suggestion should be bannable.
CLOVNEA
Scrap Iron Flotilla
Electus Matari
#12 - 2017-02-21 14:43:11 UTC
Did the Op manage to get ahead with this issue. I am also looking for a solution to this issue.
Fat Buddah
Federal Navy Academy
Gallente Federation
#13 - 2017-02-21 15:34:31 UTC  |  Edited by: Fat Buddah
ESI is the interface that recieves your request and delivers what's asked. CCP going from XML to ESI is equivalent to a business going from mail order to Amazon. Think of Python, php, c++ etc as browsers you use to order stuff from Amazon.
When you say you wantnto learn ESI you actually mean how do I call and receive stuff from ESI using the language X.
In your case the tool you are familiar with is spreadsheets. Unfortunately, there isn't an easy method to interact with ESI from a worksheet at the moment. There was an ESI proxy service for a short period (several threads down) that worked with spreadsheets, but it's been taken down.
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#14 - 2017-02-21 17:29:08 UTC
VBA is a total PITA wrt to JSON

Of course, there's always XLWings, if you want to do python on excel Big smile

There's a video from fanfest with a little detail on how to do it.

https://www.youtube.com/watch?v=T4osYwR61Ak

I'm talking about CREST in it, but ESI is pretty close.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Haile Korhal
Professional Amateurs
#15 - 2017-03-13 21:14:10 UTC
So, to migrate to ESI from XMLimport we need to learn Python? I'm afraid scripting is well above my level of comprehension. I can fumble my way through reading it because I learned VBA and C++ 6.0 in high school (and played with C#, LUA, and FORTH in Minecraft), but that was all so long ago... :(

I lean heavily on your knowledge and website Steve, save us from the new dark-age!

Egregious Spreadsheet Services - For Spreadsheets as a Service to businesses, corporations, and higher, look no further!
Blacksmoke16
Resilience.
#16 - 2017-03-14 00:30:07 UTC
Not required no. Using ESI is a bit more in dept for sure than just doing importxml(endpointURL,xpath) but it is also much more useful. I know for sure the public endpoints ( and probably even the private ones somehow) can be used in google sheets. However to get them you would have to use the scripting feature of sheets, by either creating one yourself or using an already made one like Nuadi's market one.

salacious necrosis
Garoun Investment Bank
Gallente Federation
#17 - 2017-03-14 01:48:00 UTC
Hmm, maybe I should reconsider opening up my site after all. I came up with one solution to this problem here:

https://esi-proxy.orbital.enterprises/#/main

That solves the OAuth problem for you and lets you use XML style API keys with the ESI. I suppose I could rate limit or otherwise prevent people from doing bad things.

Once you've done that, something like Google Sheets boils down to JSON parsing which isn't too bad. I DID release my code as open source here:

https://github.com/OrbitalEnterprises/orbital-esi-proxy

But I'm guessing the OP doesn't want to run their own site.

Use EveKit ! - Tools for EVE Online 3rd party development
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#18 - 2017-03-14 12:48:36 UTC
salacious necrosis wrote:
Hmm, maybe I should reconsider opening up my site after all. I came up with one solution to this problem here:

https://esi-proxy.orbital.enterprises/#/main

That solves the OAuth problem for you and lets you use XML style API keys with the ESI. I suppose I could rate limit or otherwise prevent people from doing bad things.

Once you've done that, something like Google Sheets boils down to JSON parsing which isn't too bad. I DID release my code as open source here:

https://github.com/OrbitalEnterprises/orbital-esi-proxy

But I'm guessing the OP doesn't want to run their own site.



The biggest problem with a proxy, is that the person running the proxy is responsible for all activity that passes through it.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#19 - 2017-03-14 12:50:15 UTC
Haile Korhal wrote:
So, to migrate to ESI from XMLimport we need to learn Python? I'm afraid scripting is well above my level of comprehension. I can fumble my way through reading it because I learned VBA and C++ 6.0 in high school (and played with C#, LUA, and FORTH in Minecraft), but that was all so long ago... :(

I lean heavily on your knowledge and website Steve, save us from the new dark-age!



There's no _requirement_ to learn Python. It's just a good idea Blink

It's possible to do things in VBA, but it's a pain to interact with. If you're just working with public data, powerquery can do it fine, but it has problems with doing the authentication step. apparently it can do it, but I've yet to get it working.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Zifrian
Federal Defense Union
Gallente Federation
#20 - 2017-03-14 15:22:26 UTC
BTW I brought up VBA just because it's free and it's in Excel. It's basically a slim version of VB6 and it sucks, but it's better than nothing if you don't know how to do anything else. You can hack quite a bit in VBA and get stuff to work.

I use VB.Net because I'm not a programmer anymore and I don't need/want to learn a new language that I'll never use anyway. Plus I can do almost everything in VB.Net that you could do in C#. The differences are very minor now and it makes VB a great hobby language.

That said, I still haven't tried to use ESI in IPH. I don't know if I will either Sad

Maximze your Industry Potential! - Download EVE Isk per Hour!

Import CCP's SDE - EVE SDE Database Builder
12Next page