These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

[EveLib] A .NET library for EveXML, CREST, EveCentral, and more

First post
Author
Brain Gehirn
Macabre Votum
Northern Coalition.
#401 - 2016-05-05 17:35:02 UTC
Aio Kaze wrote:
The LP store endpoint would be very useful as I've been wanting to put together an LP profit calculator and don't fancy entering thousands of LP/ISK/item requirements by hand.



afaik there is no such endpoint
Mr Mac
Dark Goliath
#402 - 2016-05-05 17:56:08 UTC
Brain Gehirn wrote:
Aio Kaze wrote:
The LP store endpoint would be very useful as I've been wanting to put together an LP profit calculator and don't fancy entering thousands of LP/ISK/item requirements by hand.



afaik there is no such endpoint

You can get LP store from test crest
Brain Gehirn
Macabre Votum
Northern Coalition.
#403 - 2016-05-05 23:04:34 UTC
Awesome, just saw it.
Btw its working in the TQ CREST too, not just test.
Aeon Haginen
#404 - 2016-05-11 22:49:23 UTC
Hi.
I must admit, that your lib is a great work!

But... I'm getting a lot of "Safe handle has been closed" exceptions when using crest.

I'm using only one instance of EveCrest.
When using crest.GetRoot().Query(r=>r.Regions), its like 5 of 6 times, when i get that exception.
On market queries, i get it in about 1% of queries.

My code is:
            var allRegions =
                crest.GetRoot()
                    .Query(r => r.Regions)
                    .Query(r => r.Items)
                    .OrderBy(item => Helpers.GetRegionOrder((int)item.Id));


Stack trace is:
   at System.Runtime.InteropServices.SafeHandle.DangerousAddRef(Boolean& success)
   at System.StubHelpers.StubHelpers.SafeHandleAddRef(SafeHandle pHandle, Boolean& success)
   at System.Security.Cryptography.Utils.HashData(SafeHashHandle hHash, Byte[] data, Int32 cbData, Int32 ibStart, Int32 cbSize)
   at System.Security.Cryptography.SHA1CryptoServiceProvider.HashCore(Byte[] rgb, Int32 ibStart, Int32 cbSize)
   at System.Security.Cryptography.HashAlgorithm.ComputeHash(Byte[] buffer)
   at eZet.EveLib.Core.Cache.EveLibFileCache.getHash(Uri uri)
   at eZet.EveLib.Core.Cache.EveLibFileCache.< StoreAsync >d__13.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at eZet.EveLib.EveCrestModule.RequestHandlers.CachedCrestRequestHandler.< GetAsync >d__53`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at eZet.EveLib.EveCrestModule.EveCrest.< getAsync >d__121`1.MoveNext()


Do you think you can look into it, or give me a hint, what should i do to not get this exception?
Thanks.
Icahmura Hasaki
Perkone
Caldari State
#405 - 2016-05-12 12:30:22 UTC
I've updated Core, fixing a bug that would sometimes throw ObjectDisposedException when attempting to write the cache.
https://www.nuget.org/packages/eZet.EveLib.Core/3.0.8

Developer of EveLib and EveAuthUtility

Icahmura Hasaki
Perkone
Caldari State
#406 - 2016-05-12 13:44:04 UTC
Added support for LP stores as requested, and also NpcCorporationCollection and Stations.
https://www.nuget.org/packages/eZet.EveLib.EveCrest/3.4.3

Developer of EveLib and EveAuthUtility

Aeon Haginen
#407 - 2016-05-13 13:04:28 UTC
Icahmura Hasaki wrote:
I've updated Core, fixing a bug that would sometimes throw ObjectDisposedException when attempting to write the cache.
https://www.nuget.org/packages/eZet.EveLib.Core/3.0.8


Thx, now its working like expected.


I have one question though... are you planning on implementing some rate limit mechanizm instead of limiting concurrent requests with semaphore?
Icahmura Hasaki
Perkone
Caldari State
#408 - 2016-05-13 14:11:16 UTC
Aeon Haginen wrote:
Icahmura Hasaki wrote:
I've updated Core, fixing a bug that would sometimes throw ObjectDisposedException when attempting to write the cache.
https://www.nuget.org/packages/eZet.EveLib.Core/3.0.8


Thx, now its working like expected.


I have one question though... are you planning on implementing some rate limit mechanizm instead of limiting concurrent requests with semaphore?


No plans, no. Why, is it causing any problems ?

Developer of EveLib and EveAuthUtility

Aeon Haginen
#409 - 2016-05-13 14:51:20 UTC
Icahmura Hasaki wrote:
Aeon Haginen wrote:
Icahmura Hasaki wrote:
I've updated Core, fixing a bug that would sometimes throw ObjectDisposedException when attempting to write the cache.
https://www.nuget.org/packages/eZet.EveLib.Core/3.0.8


Thx, now its working like expected.


I have one question though... are you planning on implementing some rate limit mechanizm instead of limiting concurrent requests with semaphore?


No plans, no. Why, is it causing any problems ?


No, but when i'm doing parallel quering, i dont want end up being banned by CCP when i end up doing 100req/sec. It can happen wery offen, as lot of queries that im doing are parallel and are less than 100ms. (like for example detail information about regions.. 100 reqs, done mostly in about 1 or 2 sec). as those requests are done inside your lib, im not able to limit it.
i was able to limit queries to your lib for 10q/sec, but i cant control how many real requests (excluding cache hits) are being made from your lib to crest.
Icahmura Hasaki
Perkone
Caldari State
#410 - 2016-05-13 14:58:14 UTC
Aeon Haginen wrote:
Icahmura Hasaki wrote:
Aeon Haginen wrote:
Icahmura Hasaki wrote:
I've updated Core, fixing a bug that would sometimes throw ObjectDisposedException when attempting to write the cache.
https://www.nuget.org/packages/eZet.EveLib.Core/3.0.8


Thx, now its working like expected.


I have one question though... are you planning on implementing some rate limit mechanizm instead of limiting concurrent requests with semaphore?


No plans, no. Why, is it causing any problems ?


No, but when i'm doing parallel quering, i dont want end up being banned by CCP when i end up doing 100req/sec. It can happen wery offen, as lot of queries that im doing are parallel and are less than 100ms. (like for example detail information about regions.. 100 reqs, done mostly in about 1 or 2 sec). as those requests are done inside your lib, im not able to limit it.
i was able to limit queries to your lib for 10q/sec, but i cant control how many real requests (excluding cache hits) are being made from your lib to crest.


You can set the amount of concurrent requests you allow through EveCrest.RequestHandler.AuthedMaxConcurrentRequests. And as far as I know 100 req/sec is roughly what is allowed, which is why the default value is what it is.

Developer of EveLib and EveAuthUtility

Brain Gehirn
Macabre Votum
Northern Coalition.
#411 - 2016-05-13 15:00:13 UTC
What are you doing to work with parallel querys?
I'm **** with parallel things and was having issues (mostly with the cache).

When I need to update market order information for like 100 products.. it takes like 25/30 seconds. I have tried the Parallel.ForEach but then got stuck with the cache issue. (last version gave me some errors but this new version is giving a lot more, so I just gave up on the parallel thing)
Icahmura Hasaki
Perkone
Caldari State
#412 - 2016-05-13 15:16:53 UTC
Brain Gehirn wrote:
What are you doing to work with parallel querys?
I'm **** with parallel things and was having issues (mostly with the cache).

When I need to update market order information for like 100 products.. it takes like 25/30 seconds. I have tried the Parallel.ForEach but then got stuck with the cache issue. (last version gave me some errors but this new version is giving a lot more, so I just gave up on the parallel thing)


The library is built from the ground up using the task API and is how it should be used, ideally. There are async versions of all methods. Asynchronicity is not the same as parallelism, but in this case where most of the time is spent waiting for a response, parallelism doesn't make much sense.

With that said, what errors are you getting ?

Developer of EveLib and EveAuthUtility

Brain Gehirn
Macabre Votum
Northern Coalition.
#413 - 2016-05-13 15:28:26 UTC
Yeah I need to update my skills as I have been falling a bit behind... haven't had time to read/learn about it yet.

Most of issues apeeared to be around cache files being used already. But as you said its probably due the **** implementation that I have made.

I will try to learn about it this weekend and will also try to adjust my code accordingly
Icahmura Hasaki
Perkone
Caldari State
#414 - 2016-05-13 15:36:37 UTC
Brain Gehirn wrote:
Yeah I need to update my skills as I have been falling a bit behind... haven't had time to read/learn about it yet.

Most of issues apeeared to be around cache files being used already. But as you said its probably due the **** implementation that I have made.

I will try to learn about it this weekend and will also try to adjust my code accordingly


If you give me some more code examples I'll have a look again. But, if you want better performance, use the async methods, that is, the methods that end with Async. Using these will take care of everything in regards to "parallelism", granted you will probably have to make some changes to your code. I could possibly help if you've got examples.

One would think the latest updates would help with your way of doing it though, as access to the cache is being controlled by locks now.

Developer of EveLib and EveAuthUtility

Aeon Haginen
#415 - 2016-05-13 19:49:47 UTC  |  Edited by: Aeon Haginen
Icahmura Hasaki wrote:
Aeon Haginen wrote:
Icahmura Hasaki wrote:
Aeon Haginen wrote:
Icahmura Hasaki wrote:
I've updated Core, fixing a bug that would sometimes throw ObjectDisposedException when attempting to write the cache.
https://www.nuget.org/packages/eZet.EveLib.Core/3.0.8


Thx, now its working like expected.


I have one question though... are you planning on implementing some rate limit mechanizm instead of limiting concurrent requests with semaphore?


No plans, no. Why, is it causing any problems ?


No, but when i'm doing parallel quering, i dont want end up being banned by CCP when i end up doing 100req/sec. It can happen wery offen, as lot of queries that im doing are parallel and are less than 100ms. (like for example detail information about regions.. 100 reqs, done mostly in about 1 or 2 sec). as those requests are done inside your lib, im not able to limit it.
i was able to limit queries to your lib for 10q/sec, but i cant control how many real requests (excluding cache hits) are being made from your lib to crest.


You can set the amount of concurrent requests you allow through EveCrest.RequestHandler.AuthedMaxConcurrentRequests. And as far as I know 100 req/sec is roughly what is allowed, which is why the default value is what it is.


ah, they updated limits:
General Rate Limit: 150 requests per second
Burst Size: 400
Concurrent Connections: 20 (as i remember, there were 2 separate limits, one for authed and one for unauthed requests)

now my app works great. currently about 20 req/s.
Icahmura Hasaki
Perkone
Caldari State
#416 - 2016-05-14 14:18:03 UTC
A small project I've been working on lately to reduce the maintenance efforts required to keep up with crest changes, here's an attempt at a dynamic crest library for .net.

https://www.nuget.org/packages/eZet.EveLib.DynamicCrest

In many ways it's similar to the EveCrest library, at least syntactically, but also very different.

It currently supports GetAsync and PostAsync, Put and Delete are not implemented yet. GetAsync is equal to QueryAsync in EveCrest. The major difference is, all methods return Expando objects, which is a custom implementation of DynamicObject, which support late binding. This means it offers no static or compile time type checking, but also automatically supports every crest resource right out of the gate, and any changes will be reflected immediately.

Sample use:

var crest = new DynamicCrest();
var root = await crest.GetAsync(crest.Host);
var alliance = await (await root.GetAsync(r => r.alliances)).GetAsync(a => a.First(o => o.id == 99000967));


As you can see, it's very similar to how EveCrest is used. It already supports automatic pagination when querying collections, and all the other good stuff. One difference is that the property names are now strictly camelCased, directly mirroring the json response. So what would be r.Alliances in EveCrest is now r.alliances.

Basic support for POST requests are also available:

        public async Task AddWaypoint() {
            var character = await (await (await crest.GetAsync(crest.Host)).GetAsync(r => r.decode)).GetAsync(r => r.character);
            var system = (await(await crest.GetAsync(crest.Host)).GetAsync(r => r.systems)).First();
            or //var system = new Resource(30000142, "http://crest.regner.dev/solarsystems/30000142/");
            var wp = new AutopilotWaypoint(system);
            or //wp.SolarSystem = system;
            await crest.PostAsync(character["waypoints"].href, wp);
        }


If you need access to an endpoint that isn't yet supported in EveCrest, this can be used to access anything made available through crest. To find which properties are available, you can enumerate the Properties property on the Expando object, which provides an IDictionary[string, object] interface to all the properties.

Keep in mind that this is an early preview.
In time I hope to have this replace EveCrest, since the amount of maintenance needed to keep up with CREST changes is a bit much.

Developer of EveLib and EveAuthUtility

Dread Griffin
Pator Tech School
Minmatar Republic
#417 - 2016-05-27 16:53:13 UTC
Thanks for this tool, it's very helpful!

I've successfully used EveAuth to get my access and refresh tokens. Next, I create the EveCrest object using my refresh token and encoded key. All great up to this point.

However, when I run GetRoot() on the crest object in authenticated mode it doesn't seem to include anything extra for the scopes I authenticated with. Maybe this is working as expected and i'm just not understanding.

I after reading about RESTful practices, I assumed the root would return additional endpoints if an access token was used. Is this not the case?
Icahmura Hasaki
Perkone
Caldari State
#418 - 2016-05-28 15:24:06 UTC
Dread Griffin wrote:
Thanks for this tool, it's very helpful!

I've successfully used EveAuth to get my access and refresh tokens. Next, I create the EveCrest object using my refresh token and encoded key. All great up to this point.

However, when I run GetRoot() on the crest object in authenticated mode it doesn't seem to include anything extra for the scopes I authenticated with. Maybe this is working as expected and i'm just not understanding.

I after reading about RESTful practices, I assumed the root would return additional endpoints if an access token was used. Is this not the case?


Root is the same, unauthorized requests will be denied upon performing the request.

Developer of EveLib and EveAuthUtility

Dread Griffin
Pator Tech School
Minmatar Republic
#419 - 2016-05-30 05:56:06 UTC
Icahmura Hasaki wrote:
Root is the same, unauthorized requests will be denied upon performing the request.


Ok, thanks for the reply. Then I guess I'm missing how I'm supposed to use EVELib to access endpoints requiring authentication. I thought we had to get the root and then navigate from there using querys and loads.
Icahmura Hasaki
Perkone
Caldari State
#420 - 2016-05-31 08:28:11 UTC
Dread Griffin wrote:
Icahmura Hasaki wrote:
Root is the same, unauthorized requests will be denied upon performing the request.


Ok, thanks for the reply. Then I guess I'm missing how I'm supposed to use EVELib to access endpoints requiring authentication. I thought we had to get the root and then navigate from there using querys and loads.


Well, they will only be denied if the request is unauthorized. If you authenticate with the SSO you should be good ? And yeah, you are supposed to go through the root. All I'm saying is the root doesn't change based on authentication, the authorization check is performed upon requesting an endpoint that requires authentication, the root is public.

Developer of EveLib and EveAuthUtility