* Evaluation service interface description: [http://www.wf4ever-project.org/wiki/display/docs/RO+checklist+evaluation+API]
* RO dereferencing notes: [http://www.wf4ever-project.org/wiki/display/docs/RO+dereferencing]
h2. Selection of ROs examples
The first example that has been used in order to test the integration has been the Simple-Requirements RO. This RO has been evaluated by using the different checklists that can be found in the minim included in the RO and showing the results in the webapp that performs the calls to the evaluation service.
Simple-requirements \[5\] (is simple RO and focused on completeness): "An RO based on a simple shell-script workflow with MINIM descriptions for evaluating basic notions of replayability (inputs and script aggregated and required software present), reviewability (script, inputs and outputs aggregated), repeatability (script, inputs, outputs aggregated and software present)".
The next proposed examples are:
* Protein discovery \[6\](WF74): "A workflow example that has been used in many demonstrations is a Protein Discovery workflow. It follows the basic text mining procedure to produce proteins that were found together with the terms in the input query in the abstracts of biomedical papers."
* Workflow Repeatability \[7\](WF74-repeatable-fail): "This RO is based in the repeatability concept that focuses on the ability of "repeating" an experiment even if the results obtained are different." The RO is a copy of the WF74 with minor changes tom make it fail the repeatability test. In this showcase, this notion has been partially explored - see _Outstanding issues_ below.
h2. MINIM web services
The description of the minim checklist evaluation web service interface can be found at ?[docs:RO checklist evaluation API] \[1\]. It uses code developed as part of RO-manager, invoked via a web interface constructed using the Pylons Pyramid web framework \[4\]. For the purposes of integration testing, the evaluation service was deployed on a development system at [http://andros.zoo.ox.ac.uk:8080/], with copies of the RO data to be evaluated at [http://andros.zoo.ox.ac.uk/workspace] and/or [http://andros.zoo.ox.ac.uk/aleix]. The Sample RO data is a direct checkout of the github repository at [https://github.com/wf4ever/ro-catalogue], served by a standard Apache httpd web server. Content negotiation as proposed in ?[docs:RO dereferencing] has not been implemented.
Progress notes:
20120508 (Tue): planning; prepare RDF example of evaluation results (?[docs:RO checklist evaluation API]) ([https://jira.man.poznan.pl/jira/browse/WFE-447])
20120509 (Wed): Initial web service based on Pyramid; content-negotiated service document; set up system for test service ([https://jira.man.poznan.pl/jira/browse/WFE-441])
20120510 (Thu): Deployed initial service on test system; delayed by getting access enabled through firewall; OK now ([http://andros.zoo.ox.ac.uk:8080/]); create service document with content negotiation for testing (NOTE: accept headers must use lowercase content-types, text/html, text/turtle or application/rdf+xml); confirmed temporary mechanism to pull manifest from RO in RODL
20120511 (Fri): Basic web server deployed; dummy evaluation service deployed; uri template expansion service deployed; tried to push simple-requirements example to RODL - the RO is present, manifest is accessible, but annotations and component resources are not accessible. Today: push ahead with service implementation. Blockers: RODL access problems (not really a blocker, but an inconvenience - will probably get picked up later).
20120515 (Tue): Refactoring of checklist evaluation service to access web-based ROs is done. Next step is to plug this into the web framework, which I've started (maybe about 30% done?).
20120516, \-17 (Wed, Thu): continued refactoring. Demo RO evaluation service deployed and running at [http://andros.zoo.ox.ac.uk:8080/].
20120518 (Fri): started looking at RO repeatability testing; agreed that we needed to link a "re-run" RO to a "calibration RO" for comparison purposes; investigated possible designs for content-match detection; working with Aleix to deploy sample wf74 RO, with variations to demonstrate repeatablity failure; requested access to create purl.org/wf4ever/roeval namespace URI; investigate possible use of ni: for change detection; look into Pyramid content negotiation problems
20120521 (Mon): Work with Aleix to resolve not-repeatable test problem (wfdesc encoding issues); added material to wiki; started on proposals for some technical elements of proposed future work
h2. I&A Evaluation (Runnable, Reviewable and Repeatible)
The evaluation done on the different ROs has been focused on previous work and requirements. For each purpose there is a different set of requirements that are cheked in orther to get the evaluation.
During this sprint we have been testing the RO's by using different primitives that were implemented (in that way we can try and see if they fullfill the expectations and detect new primitives that should be taken into account in order to measure other possible aspects of the RO needed to achieve a specific purpose).
The main checklists used are related to Runnability, Reviewability and Repeatability \[2\]
A +Runnable+ RO according to the used checklist is an RO that has some enveironmental requirements, such as having lpod-show and Python, and some files that need to be present, as the workflow and its inputs. Appart from those requirements, we will add the need of having the outputs in we want our RO to be R{+}epeatable+. On the other hand, if we want our RO to be +Reviewable+ we won't need to check the environmental requirements but we should have not only the workflow instance, inputs and outputs, but also the worflow provenance generated by a workflow run.
h2. How do I build a MINIM?
A description of the MINIM model can be found at \[2\] and the brief tutorial that we did for better understanding can be also found at \[3\]. Basically the MINIM model is base in three concepts:
* Constraints: they define the model to be applied for an specific purpose (e.g. runnable_RO). So it is an abstraction that layers on a minim:Model, allowing to associate an specific purpose and target resource to that model. The relation with a model is 1-1 or N-1 (multiple constraints could refer to the same Model). The check that a specific workflow in an RO is runnable, the target could be a wfdesc:Workflow within the RO, and the purpose could be "Runnable". If no target is given, the RO itself is assumed.
* Models: they define the set of requirements to be acomplished to be compliant with an specific constraint (e.g. runnable_RO_model). Each requirement is associated with a rule which is invoked to evaluate the requirement.
* Requirements: the rules used to evaluate requirements are built on a set of "primitives" which test some specific aspects related with a RO in order to produce a pass or fail outcome (e.g. isPresent/workflow-outputfiles). Some of the primitives are driven by a SPARQL query that is executed over the merged RO manifest and annotations.
The next draw show the relation between these three concepts.
!uml.png|border=1,width=402,height=232!
Key elements of the tutorial material have been added to the service interface description at ?[docs:RO checklist evaluation API].
h2. Towards Integration and documentation of services
The focus for integration has been the checklist evaluation service implemented by Oxford as part of RO-manager, and to be used as part of a higher-level RO Integrity and Authenticity evaluation service for researchers that is being implemented by iSOCO. The intended interaction between I&A components is illustrated below.
!WP4-technical-integration.png|border=1!
...