These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

iveeCore: The PHP engine for industrial activities and CREST library

Author
Aineko Macx
#1 - 2016-04-24 17:12:41 UTC
The previous thread closed, so I'm opening this one to announce the release of iveeCore 3.0!

Purpose and target audience

The goal of this project is to provide its users with a simple but powerful API to calculate industrial activities in EVE Online and get data such as bill of materials, activity costs and profits or skill requirements. iveeCore helps developers to quickly prototype their own scripts or develop full blown (web) applications without having to directly deal with the complexities and quirks of EvE's Static Data Export and CREST API.

iveeCore will likely be most useful for developers with at least basic PHP knowledge creating their own industry related or CREST powered tools.

These are a few example questions that can be answered with a few lines of code using iveeCore:

  • "What is the profit of buying this item in Jita and selling in Dodixie?"
  • "What do these items reprocess to? And what is it's volume sell value in Jita?"
  • "How much does this EFT fit cost? How much is this scanned cargo worth?"
  • "What is the job cost for building this item in this system? And if built in a POS in low sec?"
  • "What is the total bill of materials and the minimum required skills to build these Jump Freighters?"
  • "Is it more profitable to build the components myself or buy off the market?"
  • "What is the total profit for copying this Blueprint, inventing with it, building from the resulting T2 BPC and selling the result?"
  • "How do different Decryptors affect ISK/hour?"
  • "How do Blueprint ME levels impact capital ship manufacturing profits?"
  • "How much is a month of Unrefined Ferrofluid Reaction worth?"

The iveeCrest portion of the engine allows you to do things like:

  • Access CREST endpoints for universe data (regions, constellations, systems, planets), inventory (types, groups, categories), live market data (orders and history), alliances, sovereignty, wars & killmails, alliance tournaments and industry related data.
  • Read and write to the private character endpoints for fittings, contacts and navigation.


Technical features

  • An API that strives to be "good", with high power-to-weight ratio
  • Strong object oriented design and class model for inventory types
  • Classes for representing manufacturing, copying, T2 & T3 invention, research and reaction activities, with recursive component building
  • Realistic price estimation (based on CREST market order data), profit calculation & DB persistence
  • Parsers for EFT-style and EvE XML ship fittings descriptions as well as cargo and ship scanning results
  • Caching support for Memcached or Redis (via PhpRedis)
  • Extensible via configurable subclassing
  • A well documented and mostly PSR compliant codebase
  • CLI tool for automated bulk market data updates

The CREST portion of the engine (formerly the iveeCrest library) can be used in conjunction with iveeCore or independently. Some of its features include:

  • Aims to be feature-complete and do things "the CREST way" as much as possible
  • Support for public and authenticated CREST including write access
  • Object oriented model for navigating the CREST endpoint tree, mapping CREST responses to specific classes
  • Gathering and re-indexing of multipage responses
  • Supports parallel GET requests with asynchronous response processing
  • Multilayer cache design
  • Includes a self-contained web-script to retrieve a refresh token


DOWNLOAD
The repository with more information and documentation can be found here
https://github.com/aineko-m/iveeCore

Packagist:
https://packagist.org/packages/aineko-m/ivee-core

Feedback welcome!
Aineko Macx
#2 - 2016-04-24 17:17:07 UTC  |  Edited by: Aineko Macx
I've been playing catch up with the changes to CREST for a while, also being delayed by the huge iveeCrest refactoring. I decided to release now, shortly before citadels hit, because there are no post-citadel changes in the code and I didn't want to delay the release further to add them. A smaller release will likely follow shortly after to add whatever changes are necessary.

The release notes for 3.0.0:

iveeCrest has been thoroughly refactored:

  • Typed responses have been implemented, so depending on the response content types, dedicated classes get instantiated for them.
  • EndpointHandler has been removed. All of its functionality is now offered by the response classes.
  • Multiple authentication scopes are now supported.
  • Write access to CREST is supported.
  • Non-authenticated CREST endpoints can now be accessed without the need to set up the application in the Eve developer backend.
  • Collections are now gathered in parallel for higher performance.
  • The getrefreshtoken PHP web script now supports the selection of multiple authentication scopes.


Other improvements include:

  • The iveeCore CREST updater has been improved, all CREST data can now be updated on-the-fly if it is too old.
  • Solar system industry indices have been moved to their own class, SytemIndustryIndices.
  • In InventionProcessData the methods available in a regular and a "success" variant have been renamed to "attempt" and regular, respectively. This avoids specialcasing for handling ProcessData object trees and is less error prone.
  • Rewritten documentation.
  • The codebase is now more PSR-2 compliant.
  • Improved PHPUnit tests.
  • Various bug fixes and removed redundancies.
Aineko Macx
#3 - 2016-04-28 20:22:01 UTC
Alright, 3.0.1 is out with the first set of changes to adapt to the Citadel release. Note that CREST is currently still exhibiting all sorts of ailments. In the coming weeks I'll be adding functionality to use all those new CREST capabilities.

Release notes for 3.0.1:

  • Removed the distinction between public and authenticated CREST domains. Minor iveeCrest\Client API change.
  • Implemented and started using the combined buy & sell market order CREST endpoint.
  • Added all new authentication scopes to getrefreshtoken.php
  • Minor fixes.
Jugular Vein
Hoover Inc.
Pandemic Legion
#4 - 2016-05-26 20:47:16 UTC
Thanks for this.
Peter Ford
Fat Mustache
Beanstalk Inc.
#5 - 2016-06-15 07:51:08 UTC
Hello

I posted an issue in github related 2016-06-14 EvE patch. Below copy of an issue.

https://github.com/aineko-m/iveeCore/issues/7

Quote:
Hello,

I'm using iveeCore inside Symfony 1.4 application.

After 2016-06-14 patch (https://developers.eveonline.com/blog/article/crest-renovations-2016-06) I got error:

PHP Fatal error: Call to undefined method iveeCrest\Responses\BaseResponse::getRegionCollection() in /mnt/data/apache/............../lib/task/eveCrestTask.class.php on line 115

I know that many urls are wrong after patch.

But why method is undefined? I Assume that is problem with inheritance.

I know that baseUrl is in Config class. Where are defined other parts of urls?

Are you planning update of script?


o/
Aineko Macx
#6 - 2016-06-15 08:28:41 UTC
Yes, the problem is due to the CREST API breakage from yesterdays changes. I'm working on an update. The current dev branch should already be fixed.
Aineko Macx
#7 - 2016-06-15 14:11:38 UTC
After I took a development break due to RL, I'm back with the 3.0.3 release. The most important changes are the fixes for the recent CREST API breakage and support for the combined region-wide market order collection. The latter allows you to run complete eve-wide price update batches in about 12 minutes Cool , but it is not enabled by default as it is quite memory intensive.

Version 3.0.3 - June 15th, 2016

This release adds support for new CREST endpoints and fixes issues that cropped up.

  • Fixes for the June 14th CREST update.
  • Added MarketOrderCollectionSlim for the new combined region-wide market order collection endpoint.
  • The batch price update can use MarketOrderCollectionSlim, but since it is a memory intensive feature, it must manually be enabled in Config.
  • Added classes for Stargate, Moon, Station, InsurancePrices, LoyaltyPoints, LoyaltyStoreOffers, NPCCorporations as well as Opportunity endpoints.
  • Remove redundant CREST attributes from JSON ("_str") via regex before decode, which is much faster.
  • Adapt getrefreshtoken to new login domain, warn about requesting too many auth scopes.
  • A number of small fixes, improvements and removal of obsolete stuff.


I hadn't announced the 3.0.2 release, so here are the release notes for completeness:

Version 3.0.2 - April 30th, 2016

Release with bugfixes and minor improvements. The initial CREST issues post-Citadel release have been fixed by CCP. The current SDE is still missing industry data for some of the new items.

  • Use new broker fee and transaction tax formulas.
  • When no global price data is available for calculating product base cost (for instance new item) in Blueprint activities, use Jita buy price as approximation.
  • Auto retry logic in CurlWrapper::asyncMultiget(), with abort on too many errors; also stop using the error callback. Some exception handling improvements.
  • Fix multipage gathering for Collections where pageCount=0.
  • Don't forget to mention DoctrineCacheWrapper in documentation.
Peter Ford
Fat Mustache
Beanstalk Inc.
#8 - 2016-07-07 07:29:40 UTC  |  Edited by: Peter Ford
Hello Aineko Macx,

I see new problem.
In last patch 2016-07-05 I think CCP added orders from citadels.

In my code I got exception:
Station ID=1021159407897 not found

Others also noticed this problem:
https://forums.eveonline.com/default.aspx?g=posts&t=486848&find=unread

BR
Peter Ford

P.S.
Can I donate you with some ISK?
Dragonaire
Here there be Dragons
#9 - 2016-07-08 00:36:38 UTC
I don't know the code in iveeCore but tend to keep everything Eve related as strings except when I have to use a number just because of PHP's lack of/spotty record for 64bit integers until very recently. Also better to treat anything ending in ID from CCP like 20 digit unsigned integer in DB until proven otherwise I've had many painful encounters during work with XML APIs when I've ignored that in Yapeal P

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Aineko Macx
#10 - 2016-07-11 06:17:36 UTC  |  Edited by: Aineko Macx
Peter Ford wrote:
In my code I got exception:
Station ID=1021159407897 not found

Hi Peter, sorry for the delay. What exactly were you calling when you ran into that? Running the batch price update does not trigger the exception. And if you are trying $root->getStation(1021159407897) then well, the problem is that CREST does not provide Citadel data. EDIT: I won't say no to ISK Cool

Dragonaire wrote:
because of PHP's lack of/spotty record for 64bit integers until very recently

64 bit PHP has been around for a long time and has been a requirement for iveeCore since the beginning and I never had issues tbh.
Aineko Macx
#11 - 2016-09-25 07:45:21 UTC
RL is keeping me from working more on iveeCore, but here's a small update:

Version 3.0.4 - September 25th, 2016


  • Added Corporation and CorporationStructuresCollection response classes.
  • Minor changes that make iveeCore compatible with PHP 7.
  • Reenable tournament scene data tests as CCP fixed the endpoints.
  • Added new authentication scopes to getrefreshtoken script.
  • Improved the logic behind the "huge" batch price update.