Meeting via Skype conference call on 28th of November 2006

From Wiki

The meeting was held with Skype conference call on 28th of November 2006.

Contents 1 Schedule and attendance 2 Status of Task 7.5 3 Ongoing work in Task 7.5 3.1 Evaluation of Collaboration in a Digital Library Environment 3.2 Logging Scheme within INEX 4 TEL Evaluation 5 Misc

Schedule and attendance

• The schedule of the conference call meeting was
  • Status of Task 7.5
  • Task 7.5
  • TEL
  • Misc

• Attendance:
  • Preben Hansen
  • Hanne Albrechtsen
  • Giannis Tsakonas
  • Claus-Peter Klas
  • Hungary was absent with apologies.

Status of Task 7.5

• Peter gave a short overview of the work done incl.
  • JCDL paper
  • ECDL paper
  • Integration of Daffodil with DELOS DLMS
  • Preparing INEX evaluation from technical point of Daffodil
  • Integration of National Libraries into the special TEL-Daffodil prototype

Ongoing work in Task 7.5

Evaluation of Collaboration in a Digital Library Environment

Giannis is preparing an evaluation on collaboration in a DL environment. The members started a high-quality discussion about the planed evaluation. It was questioned if the evaluation should be either:

• usability test
• a real evaluation

There was a discussion about the assumptions of the evaluation which are:

• Usage of Daffodil and which other tools?
• How, in a quantitative analysis, they are using it?
• Data circulation: How are people sharing personal thoughts. Are they afraid, do they trust the system; What about security.
• What are the roles of users and how does it affect the usage? Affection of Interaction: Is the shared information more formal, if the supervisor is addressed?

Different possible evaluation scenarios were discussed like:

• Exploratory evaluation with semi-controlled groups, with the same task
  • Intro Session: Show used tool and introduce the task
  • Give a specific time schedule for the users
  • Observe and interview the users during task
  • Have post focused grouped interview.
• Controlled tests
• Qualitative & quantitative data in comparison
• Controlled test? on communication between 1 vs. many:
• Usage?: Let them do this :Are and How are they using? Loose eval with form.

Preben raised some critical questions:

• Preferences of using a tool is very personal. E.g. Wiki...vs. email: Why?
• How to monitor the interactions? Using the log schema to get times?
• Have a comparative evaluation with the same task with and without Daffodil
  • What are the outcome: better without Daffodil by time, but better knowledge with Daffodil ?
  • Total new tool to explain is very special and can confuse users
  • Example was Annotation Evaluation: A "Little Tool" made a lot of introduction information necessary for the users in order to use it. Daffodil in all its mighty is "too much". Peter stated that the concept of perspectives can lower the complexity for the user and can so focus the evaluation to the needed tools.
  • There is also the need of a pilot evaluation.

Hanne mentioned, that the DILAS annotation evaluation was actually a collaborative evaluation.

First plans of Giannis were:

1. Setup experiment
2. Methodology: use naturalistic environment

The evaluation rely on the Logging Scheme, since they can not observe the user all the time in a naturalistic environment.

Methodology (which will be revised)

1. Pre-Interview;
2. Introduction to Daffodil; get familiar!
3. Play with Daffodil a couple of day.
4. Ask about experience of the user with Daffodil
5. Task (Log-Data only) since it is not possible to observe all the time: What task ?
6. Post-Interview

The outcome of the discussion is:

• Giannis will provide a document with the assumptions, a revised methodology etc.
• The group will comment on that document
• Whoever wants to participate can then join the actual evaluation to broader the quantitative or qualitative results.

Logging Scheme within INEX

Peter proposed also to use the INEX evaluation to review the Logging Scheme and generate statistical data.

TEL Evaluation

Clarification: TEL should be extended within FP7 to a real digital library including fulltext sources, according to Norbert. That means not a portal, but much more.

We want to have meeting in January in order to clear a lot of questions:

• What are the main issues?
  • Accessibility?
  • Usability?
  • Technical stuff?
  • Sustainability?
  • Visibility?
  • Interoperability?

• Who are the stakeholders?
• Who are the users, or which are the users, that TEL wants to attract?
  • schools and pupils
  • librarians (national libraries)
  • non-scientific people, searching for leisure

The plan is to have a comparative evaluation of the TEL vs. Daffodil userinterface based on the same data sources, the NLs.

Preben stated, that we have to make a very good job on TEL since it delivers a message to the EU commission that "DELOS will make a difference also outside of DELOS"

If user groups are more or less fixed we could following scheme:

1. Find out, what TEL is, or wants to be
2. Find out, what Daffodil is
3. With this baseline in background narrow down to a specific and interesting comparison of TEL and Daffodil on an easy & complex task.

The evaluation can not be broad, due to the limited funding, so concentration on one good point should be the main idea.

The next step is to meet with TEL and possibly NL people to get the questions answered. The meeting should be scheduled in January after 15th.

Misc

• Preben wanted to clarify the budget allocation of SICS, which i forwarded to Norbert.

• DELOS Conference Call:

  I clarified with Norbert, that the DELOS conference in February is just to sum-up the DELOS work 
  in the last three years. There will be paper presentations of the submitted papers. It is not 
  planed to do any preparation or talks about new projects, officially.