Showcase 68 - API definition and implementation

{color:#800080}{*}{_}Status: sprint in progress{_}{*}{color}

{toc:minLevel=2}

h2. Introduction

The main goal of the showcase is to make sure that all Wf4Ever services have proper APIs, in terms of being RESTful and using Linked Data. This will be done by revisiting the existing or planned APIs, improving them where necessary, performing a quick or mockup implementation and finally documenting them. The APIs will be investigated in the following order:
# Checklist API documentation update (Graham)
# Evolution API (clarify required functionality) (Piotr?, Raul?)
# Workflow transformation API, based on current taverna->RO service (Stian?)
# Annotations API to allow myExperiment to add annotations to an RO in RODL (Piotr?)
# User management API (Piotr?, Kevin?)
# Stability API (Aleix?)
# Recommender API (Rafa?)
# RO aggregation API (adding new contents, etc.) (Piotr?)

People involved: *Graham*, Piotr, Kevin, API authors.

Scheduled for: 18-29 June 2012 (confirmed at stand-up 2012-06-20)

There's a github project for collecting API descriptions and examples - [https://github.com/wf4ever/apis]

h3. Kick-off meeting agenda and notes

When: 2012-06-18, 14:30 BST / 15:30 CET

Where: Skype chat (with voice if needed)

Who: Graham, Piotr, Stian, Aleix, Rafa, Kevin (lurking)

# Set the scene - meeting goals (GK - 5 mins)
#* Meeting goal is to agree goals and methodology for sprint
#** balance common structure with flexibility and local practices
#** discuss and revise (or replace) my proposals
#* Agree goals for sprint
#** what APIs will we tackle?
#** aim to describe API *and* provide simple implementations
#** what constitutes "success" for the sprint? (e.g. 3 APIs documented and implemented)
#* Confirm sprint duration
#* Agenda bashing
# Review of REST APIs (GK - 5 mins)
#* reminder of where the focus of our work should lie
#* review background materials links
# How to document APIs? (GK intro; all discuss - 10 mins)
#* Overall structure
#* github, wiki, or something else?
#* documentation - common format or each to his/her own tools?
#** RESOLVED: Documentation in wiki; links to implementations in github or elsewhere
#** Single page (separate from showcase page) listing the APIs, with links to wherever they are created. Link API page from showcase page.
#** ACTION: GK create wiki page for API descriptions, and link from showcase (start with page under architecture, maybe move later)
# How to implement APIs? (GK intro; all discuss - 15 mins)
#* Sample services? Do we want these (or just use live deployments)? How to implement?
#** RESOLVED: each sample service implementation using implementers chosen tool; maybe live service or temporary mock-up or sample. Link to implentetaion from Showcase page.
#* Test clients? How to implement
#** Python unit tests, shell scripts, other, ...?
#** test data - as appropriate with each sample
#** RESOLVED: Client implementation by people other than service implementers
#** RESOLVED: sample client code in github API project, using implementer's tool of choice.
#** TODO: Schedule API reviews as needed (pairs or groups)
#** ACTION: GK - checklist API as example by mid-tomorrow
#** ACTION: GK rearrange api directories in GitHub
# Review sprint plan (all - 10 mins)
#* add/remove tasks
#* confirm APIs that we will tackle
#** RESOLVED: Aim to have initial draft documents for the following:
#*** Checklist (Graham)
#*** Workflow transformation (Stian)
#*** RO SRS review ; possible modularization ; covering RO access, aggregation, annotations (Piotr)
#*** Stability API (Aleix)
#*** Recommender API (Rafa)
#** Re. Evolution: Raul still working on EVO ontology - maybe bring up in second stage of sprint?
#* ACTION: each designer to come back with date for availability of initial API drafts (general expectation is for initial drafts by Wednesday)
# Next steps (all 5 mins)
#* Sprint conduct - daily "stand-up" schedule and preparation
#** 10:00/11:00 (UK/CET) daily standup - skype chat
#** Prepare with brief statement, covering: what I did yesterday; what I'm doing next; any blockers. #*\* Aim for main stand-up to take < 10 mins, with additional discussion scheduled as needed.
#* Record or link key information in showcase page - all OK with this.
#* Maintain open Skype chat channel for duration of sprint - All OK with this.
#* Confirm APIs to be tackled
#** As above; then choose 3 for initial implemtation work
# Wrap up (all 5 mins)
#* Review meeting goals - OK.
#* Review/confirm actions arising

Summary of actions:
* ACTION: GK create wiki page for API descriptions, and link from showcase. &nbsp;*Done*
* ACTION: GK rearrange template api directories in github *Done*
* ACTION: GK - checklist API as example by mid-tomorrow *Done*
* ACTION: each designer to come back with date for availability of initial API drafts

Skype chat log: ?[2012-06-18 - Skype chat log|docs:2012-06-18 - Skype chat log]

h3. Stand-ups and other discussions

* [http://www.wf4ever-project.org/wiki/display/docs/2012-06-19-SC68-standup-chat-log]
* [http://www.wf4ever-project.org/wiki/display/docs/2012-06-20-SC68-standup-chat-log]
* [http://www.wf4ever-project.org/wiki/display/docs/2012-06-21-SC68-standup]
* [http://www.wf4ever-project.org/wiki/display/docs/2012-06-22-SC68-standup]
* [http://www.wf4ever-project.org/wiki/display/docs/2012-06-22-SC68-RO-API-discussion]
* [http://www.wf4ever-project.org/wiki/display/docs/2012-06-25-SC68-standup]
* [http://www.wf4ever-project.org/wiki/display/docs/2012-06-26-SC68-standup]
* [http://www.wf4ever-project.org/wiki/display/docs/2012-06-26-SC68-RO-annotations-discussion]
* [http://www.wf4ever-project.org/wiki/display/docs/2012-06-27-SC68-standup]
* [http://www.wf4ever-project.org/wiki/display/docs/2012-06-28-SC68-standup]

* REST APIs - [http://www.wf4ever-project.org/wiki/download/attachments/3506665/REST-APIs.pdf]
* Wf4Ever architecture - [http://www.wf4ever-project.org/wiki/display/docs/Wf4Ever+Architecture]
* RO model:
** v0.1 - [http://www.wf4ever-project.org/wiki/display/docs/Research+Object+Vocabulary+Specification+v0.1]
** v0.2 (WIP) - [http://www.wf4ever-project.org/wiki/display/docs/Research+Object+Vocabulary+Specification+v0.2]
* Current user API - [http://www.wf4ever-project.org/wiki/display/docs/User+Management+2]
* Current RO SRS API - [http://www.wf4ever-project.org/wiki/display/docs/RO+SRS+interface+6+-+discussion]
* Current checklist evaluation API - [http://www.wf4ever-project.org/wiki/display/docs/RO+checklist+evaluation+API]
* Stability REST service - [Stability Evaluation|http://www.wf4ever-project.org/wiki/display/docs/Stability+Evaluation]
* [http://www.wf4ever-project.org/wiki/display/docs/2011/02/24/Research+Object+Services+%28response+plenary+session%29]
* [Wf-RO transformation service|Wf-RO transformation service]

h2. Sprint plan

NOTE: in what follows, "Describe API" includes describing or referencing the data models and formats that may be exchanged.

# Preparation: Articulate / summarize principles for REST APIs (GK) *Done.*
# Preparation: Prepare template for API description (GK) *Done*
# Preparation: Assemble links for supporting materials (GK) *Done*
# Review and discuss preparation materials (all) *Done*
# Discuss and plan framework for API mock-ups and test cases (all) *Done*
# Review priorities for tackling APIs (all) *Done*
# Draft API for checklists (GK, 2012-06-19) *Done* - [http://www.wf4ever-project.org/wiki/display/docs/RO+checklist+evaluation+API]
# Draft API for workflow transformation (Stian, 2012-06-19 @@TBC)
# Draft API for RO SRS; possible modularization of RO access, aggregation, annotations (Piotr, 2012-06-19) *Done* - [http://www.wf4ever-project.org/wiki/display/docs/RO+SRS+interface+6+-+discussion]
# Draft API for stability (Aleix, 2012-06-19) *Done* - [http://www.wf4ever-project.org/wiki/display/docs/Stability+Evaluation+API]
# Draft API for recommender (Rafa) - probably won't get done within this sprint
# Review Draft APIs in pairs or groups (all) *Done* - various group discussions; includes aggregation and annotations.
# Select 3 APIs for initial implementation (all) *Done* - focusing on just the RO API with aggregation and annotation.
# Implement RO SRS sample service (Piotr, 2012-06-29) In progress
# Implement RO SRS sample client (Graham, 2012-06-20) In progress
# Integration: sample clients working with sample services (@@TBD)

(This plan was revised on 2012-06-28 to reflect implementation focus on the RO SRS API. The original plan for for 3 distinct APIs to be implemented, but that proved to be more than we could handle within the sprint.)

h2. API description template

See:
* [https://github.com/wf4ever/apis/blob/master/api-template/template.wiki]
* [https://github.com/wf4ever/apis/blob/master/api-template/template.textile]
* [https://github.com/wf4ever/apis/blob/master/api-checklist/textile2wiki.awk]


h2. Links to API descriptions, implementations and test cases

See: [http://www.wf4ever-project.org/wiki/display/docs/Wf4Ever+service+APIs]

h3. Testing / sample framework for APIs

The testing/sample framework is intended to serve several purposes:
* to provoke some actual use of a proposed API to check that is can actually be implemented as described, and
* to provide sample implementations that supplement the descriptions
* to provide an initial basis for interoperability testing
* to provide a reference implementation for initial testing of clients that use the various services



Within the context of Wf4Ever, it is clear that different services will be implemented in different programming environments (Java+Tomcat+Wicket, Python+Pylons, Ruby+Rails are all being used for different parts of the project). As such, it is probably unreasonable to require that all service test implementations or mock-ups are implemented in a common framework. To this end, services or service mock-ups are implemented by an appropriate developer in the environment of their choice. We may want to figure out a test environment that allows these diverse services to be run alongside each other.

For client test implementations, assuming that these are generally quite simple, these may be implemented as test suites using any appropriate tools that run against the service implementations. It would be nice if this could be implemented with a view to continuous integration deployment via Jenkins or a similar system.

h2. Review and reflection

h3. Report to project meeting on 2012-06-27

_(The sprint was still in progress at the time of this report)_

Showcase 68 (APIs) - sprint review report
* See:
** http://www.wf4ever-project.org/wiki/display/docs/Showcase+68+-+API+definition+and+implementation
** http://www.wf4ever-project.org/wiki/display/docs/Wf4Ever+service+APIs

The showcase started late, and is scheduled to run until this Friday. We intend to have a reflection/review meeting tomorrow, and use the rest of the week to tidy up loose ends.

We have drafts of several APIs in various states. Original priorities have not been followed closely: we are focusing on the RO SRS API with aggregation and annotation capabilities. I plan to attempt a client implementation today. We are a bit behind on implementation because discussions about the RO API took longer that I anticipated, but we have reached a number of very important points of consensus, which are relevant to the overall project architecture. See http://www.wf4ever-project.org/wiki/display/docs/RO+SRS+interface+6+-+discussion. Arguably this (SRS) is 3 APIs in one, as it covers aggregation and annotation. I'm hoping to progress ROEVO by the end of the week, second in priority to RO SRS implementation.

h3. Notes

(Nothing noted)

h3. Sprint progress review

Progress summary:

h3. Outstanding issues

* Stability API client implementation
* Recommender API

h3. Sprint process review

The complexity and subtlety of the work needed on the RO SRS API was underestimated. The time required for this was an order of magnitude or so greater than required for specifying the checklist API that we developed previously. But we've made good and valuable progress here, which will stand us in good stead as we move forward with project-wide integration. Members of the team have found the exercise useful and informative, and that the principles and advantages of true REST APIs are a little clearer to all.

The stand-ups worked pretty well overall. On a couple of occasions, we overran our 10-minute time-slot, but I think there were genuine issues being raised. It would probably help if we could be more disciplined about dealing with communication and planning issues in the stand-up, and pushing other matters outside of that period so that people have the choice to participate or not beyond the 10-minute slot. It could help to ensure that we had everyone's full attention during the stand-up itself, if they could be confident it wouldn't drag out into the rest of their day.

There were mixed feelings about the extended discussions that took place on the open Skype channel. On one hand, we had some extremely valuable discussions, and many important issues were resolved, which we probably could not have achieved this as effectively in scheduled meetings. On the other hand, we seemed to spend a lot of time just chatting - was this the best use of our time? Using text chat is slower than voice. There was a sense in the group that the brief exchanges were useful, but that some of the extended discussions would have been better handled as a scheduled voice conference.

It was felt that in any future showcase of this nature, it would be better to be more focused in scope, and maybe have fewer people involved. For example, to showcase a single API and its implementation, and maybe also looking more to its context of use within the overall project.


h2. Related links

(Nothing noted)