What is Watson?

(1)

Finn ˚Arup Nielsen DTU Compute

Technical University of Denmark March 9, 2015

(2)

IBM Watson

Note a distinction:

IBM Watson Jeopardy version

Question-answering system with a lot a natural language processing and its own corpus taken from Wikipedia, Wiktionary, DBpedia, YAGO and a lot of other resources (Chu-Carroll et al., 2012) and IBM Watson: Beyond Jeopardy! Q&A. Specialized game control with text-to-speech.

IBM Watson commercial DTU-version

Question-passage retrieval system with a lot of natural language processing where you provide you own corpus as well as question-passages for training Watson on your corpus. Web interface for non-programmer access as well as REST API.

(3)

IBM Watson Experience Manager

DTU has access to an instance of the IBM Watson question-and- answer part of the IBM cloud ser- vices.

A graphical administration interface is available via the so-called IBM Watson Experience Manager.

Within the DTU instance there are 5 so-called “projects” available.

Each project has an associated corpus and questions linked to passages.

Only one project can be deployed at a time.

(4)

IBM Watson “Configure and train”

Steps in IBM Watson question- and-answer:

1. Upload a corpus.

2. Make questions and linked them to passages (“train”).

3. “Create corpus”.

4. Deploy corpus.

5. Test/use Watson, e.g., from the API.

(5)

IBM Watson Corpus manager

Data can come from IBM-predefined corpora, e.g., WikiVoyage or it can be uploaded by the administrator of the instance, e.g., one PDF or HTML file or several zipped together.

Watson will segment the documents, based on section heading. In HTML that will be the h1, h2 and h3 tags.

“Creating corpus”: Here is (most likely) the natural language processing taking place, where documents and questions are read and processed.

This is a somewhat slow step, e.g., a FindZebra corpus with 407 documents (6MB) takes quarter of an hour to “ingest”.

(6)

Corpora

Corpora we have tried on IBM Watson so far:

Name Domain Documents Questions

Yahoo! Answers Open (How. . . ) 2,000 (142,627) 2,000 (142,627)

FindZebra Rare diseases 363 511

DTU English site DTU 288 (0)

Beatles Beatles 269 (25)

ABBA ABBA 126 (0)

(7)

IBM Watson training

Questions and links to passages can be typed in the web interface.

The interface is relatively easy to use for a non- programmer but requires the questions typed in one by one, — each requiring several steps.

Different user roles for question entry and approval.

Watson would like 1,000 question-passage pairs, — or at least a few hundred.

(8)

IBM Watson Test

Once the corpus has been created (new) questions can be typed in the “Test” part of the Web interface.

Here the question is “diagno- sis for myoclonic and generalized tonic-clonic seizures on awaken- ing with onset in adolescence” de- rived from the FindZebra corpus and Doctor’s Dilemma.

Watson returns “Juvenile myoclonic epilepsy”, — the correct answer.

(9)

IBM Watson API

For a programmer it is easier to use the API:

> > > i m p o r t b r e d e.api.w a t s o n # My t o o l b o x w i t h a W a t s o n API

> > > w a t s o n = b r e d e.api.w a t s o n.W a t s o n()

> > > r e s p o n s e = w a t s o n.ask((’ d i a g n o s i s for m y o c l o n i c and g e n e r a l i z e d ’

’ tonic - c l o n i c s e i z u r e s on a w a k e n i n g w i t h ’

’ o n s e t in a d o l e s c e n c e ’))

> > > r e s p o n s e.e v i d e n c e l i s t[ 0 ] [’ t i t l e ’] u’ J u v e n i l e m y o c l o n i c e p i l e p s y ’

My toolbox which includes a Watson module:

https://github.com/fnielsen/brede (note the draft branch may be more developed).

For Python there is also pywatson.

(10)

IBM Watson API

Calling the question service DTU IBM Watson instance with Python:

i m p o r t json, r e q u e s t s

user, p a s s w o r d = ’ dtu ’, ’ 6 H 8 h j 3 k J J ’ # T h e s e are not the r e a l c r e d e n t i a l s q u e s t i o n = " Are you there , W a t s o n ? "

url = ’ h t t p s :// watson - w d c 0 1 . i h o s t . com / i n s t a n c e / 5 1 7 / d e e p q a / v1 / q u e s t i o n ’ r e s p o n s e = r e q u e s t s.p o s t(url=url,

h e a d e r s={’ Content - t y p e ’: ’ a p p l i c a t i o n / j s o n ’,

’ A c c e p t ’: ’ a p p l i c a t i o n / j s o n ’} ,

d a t a=j s o n.d u m p s({" q u e s t i o n ": {" q u e s t i o n T e x t ": q u e s t i o n}}) , a u t h=(user, p a s s w o r d))

‘517’ is the DTU instance. The questions are encoded in JSON and the response is also in JSON:

> > > for e v i d e n c e in r e s p o n s e.j s o n()[’ q u e s t i o n ’][’ e v i d e n c e l i s t ’]:

... p r i n t(e v i d e n c e[’ t e x t ’] [ : 8 0 ] )

T h o s e are b o t h p r e t t y p o p u l a r. To s o m e extent, a p l a i n old s t y l i n g/c u r l i n g b r u s h And if you’ re r e j e c t e d , don ’t let it b o t h e r you. You can f e e l c o n f i d e n t t h a t you F i r s t of all, you c a n n o t m a k e a l i s t of the t h i n g s you w a n t in a l i f e p a r t n e r an ...

(11)

IBM Watson API: Question analysis

Apart from the scored documents/passagers the API also returns question analysis results, e.g., slot grammar analysis, WordNet relations, question classification, . . .

With the question “Who played drums on ’Love me do’ ?” posed to the question service the following is returned (here represented in YAML):

f o c u s l i s t:

- {v a l u e: Who}

f o r m a t t e d A n s w e r: f a l s e

id: 9A 9 5 C 8 9 C D 0 C A 4 9 6 1 A 8 B F 0 E D F 9 5 1 6 0 3 E 2 i t e m s: 5

l a t l i s t:

- {v a l u e: Who} p a s s t h r u: ’ ’

p i p e l i n e i d: ’ 1 2 3 0 8 0 4 9 1 2 ’

(12)

q c l a s s l i s t:

- {v a l u e: D E S C R I P T I V E} - {v a l u e: F A C T O I D}

- {v a l u e: D E F I N I T I O N}

q u e s t i o n T e x t: Who p l a y e d d r u m s on ’ L o v e me do ’? s t a t u s: C o m p l e t e

s y n o n y m L i s t: - l e m m a: p l a y

p a r t O f S p e e c h: v e r b s y n S e t:

- n a m e: L e m m a _ E x p a n s i o n s y n o n y m:

- {i s C h o s e n: true , v a l u e: play, w e i g h t: 1 . 0 } v a l u e: p l a y e d

- l e m m a: d r u m

p a r t O f S p e e c h: n o u n s y n S e t:

(13)

- n a m e: W o r d n e t _ d r u m -noun -1 s y n o n y m:

- {i s C h o s e n: true , v a l u e: m e m b r a n o p h o n e , w e i g h t: 1 . 0 } - {i s C h o s e n: true , v a l u e: tympan , w e i g h t: 1 . 0 }

- {i s C h o s e n: true , v a l u e: drum, w e i g h t: 1 . 0 } v a l u e: d r u m s

- l e m m a: do

p a r t O f S p e e c h: v e r b s y n S e t:

- n a m e: W o r d n e t _ m a k e -verb -1 s y n o n y m:

- {i s C h o s e n: true , v a l u e: make, w e i g h t: 1 . 0 } v a l u e: do

(14)

IBM Watson API question classification

IBM Watson Jeopardy!-version had the following classes: definition, cate- gory-relation, “fill-in-the-blank”, abbreviation, puzzle, etymology, verb, translation, number, bond, multiple-choice, date (Lally et al., 2012).

API returns “qclasslist (string, optional): The container for a list of question classes that are determined by the pipeline for the final answer.”

The API seems to return somewhat different question classes:

$ p y t h o n -m b r e d e.api.w a t s o n - -y a m l " Who p l a y e d d r u m s on ’ L o v e me do ’? " \

| e g r e p -A 3 q c l a s s l i s t q c l a s s l i s t:

- {v a l u e: D E F I N I T I O N}

(15)

. . . IBM Watson API question classification

Another question classification example:

$ p y t h o n -m b r e d e.api.w a t s o n - -y a m l " W h a t d o e s the a c r o n y m ALS s t a n d for ? " \

- {v a l u e: F A C T O I D}

- {v a l u e: D E S C R I P T I V E}

And yet another:

$ p y t h o n -m b r e d e.api.w a t s o n - -y a m l " How do I c l e a n the w i n d o w of my car " \

Definition, descriptive, factoid and focusless seem to be the classes re- ported from the API.

(16)

WordNet

Inspired from (Bird et al., 2009, section 4.8, pages 169+)

(17)

. . . WordNet

Using Linux command-line interface:

$ wn g a t e h o u s e -o v e r

O v e r v i e w of n o u n g a t e h o u s e

The n o u n g a t e h o u s e has 1 s e n s e (no s e n s e s f r o m t a g g e d t e x t s) 1. g a t e h o u s e - - (a h o u s e b u i l t at a g a t e w a y; u s u a l l y the

g a t e k e e p e r’ s r e s i d e n c e )

From within Python:

> > > f r o m n l t k.c o r p u s i m p o r t w o r d n e t as wn

> > > for s y n s e t in wn.s y n s e t s(’ g a t e h o u s e ’):

... p r i n t(s y n s e t.n a m e() , s y n s e t.d e f i n i t i o n())

(u’ g a t e h o u s e . n .01 ’, u" a h o u s e b u i l t at a g a t e w a y ; u s u a l l y the g a t e k e e p e r ’ s r e s i d e n c e ")

(18)

IBM Watson API and WordNet

| e g r e p -A 10 " l e m m a : d r u m "

p a r t O f S p e e c h: n o u n s y n S e t:

- n a m e: W o r d n e t _ d r u m-noun-1 s y n o n y m:

- {i s C h o s e n: true, v a l u e: m e m b r a n o p h o n e, w e i g h t: 1 . 0 } - {i s C h o s e n: true, v a l u e: tympan, w e i g h t: 1 . 0 }

- {i s C h o s e n: true, v a l u e: drum, w e i g h t: 1 . 0 } v a l u e: d r u m s

From within Python:

> > > C o u n t e r([l e m m a for s y n s e t in wn.s y n s e t s(’ d r u m s ’)

for l e m m a in s y n s e t.l e m m a _ n a m e s( ) ] ) .m o s t _ c o m m o n(5) [(u’ d r u m ’, 9) , (u’ t y m p a n ’, 1) , (u’ m e m b r a n o p h o n e ’, 1) ,

(u’ d r u m f i s h ’, 1) , (u’ m e t a l _ d r u m ’, 1)]

(19)

IBM Watson API and part-of-speech tagging

Part-of-speech (POS) tags returned from the API:

| e g r e p -B 1 " p a r t O f S p e e c h "

- l e m m a: p l a y

p a r t O f S p e e c h: v e r b - -

p a r t O f S p e e c h: n o u n - -

- l e m m a: do

p a r t O f S p e e c h: v e r b

The POS-tags seem only to be returned for the synonyms for nouns and verbs found with WordNet: synset.pos().

Note that ‘love’ is not identified.

(20)

Python part-of-speech tagging

Finding nouns and verbs in Python with spaCy ‘somewhat equivalently’:

> > > f r o m _ _ f u t u r e _ _ i m p o r t u n i c o d e _ l i t e r a l s

> > > f r o m s p a c y.en i m p o r t E n g l i s h

> > > t e x t = " Who p l a y e d d r u m s on ’ L o v e me do ’? "

> > > nlp = E n g l i s h()

> > > t o k e n s = nlp(t e x t)

> > > [(t o k e n.orth_ , t o k e n.lemma_ , t o k e n.t a g _) for t o k e n in t o k e n s

if t o k e n.t a g _[ : 2 ] in (’ NN ’, ’ VB ’)]

[(u’ p l a y e d ’, u’ p l a y ’, u’ VBD ’) , (u’ d r u m s ’, u’ d r u m ’, u’ NNS ’) , (u’ L o v e ’, u’ l o v e ’, u’ NN ’) , (u’ do ’, u’ do ’, u’ VB ’)]

(21)

Focus and LAT

Lexical answer types (LAT):

$ p y t h o n -m b r e d e.api.w a t s o n - -y a m l " In w h a t c i t y or c o u n t r y did J o h n L e n n o n m e e t Y o k o Ono ? " | e g r e p -A 2 l a t l i s t

l a t l i s t:

- {v a l u e: c i t y} - {v a l u e: c o u n t r y}

Focus (“the part of the question that is a reference to the answer”, (Lally et al., 2012))

$ p y t h o n -m b r e d e.api.w a t s o n - -y a m l " In w h a t c i t y or c o u n t r y did J o h n L e n n o n m e e t Y o k o Ono ? " | e g r e p -A 1 f o c u s

f o c u s l i s t:

- {v a l u e: w h a t c i t y or c o u n t r y}

(22)

Focus and LAT

Results returned from the API based on a query sentence:

Sentence Focus LAT QClass

How do I wash my car? how — descriptive, factoid

Ability to probe to the bone in a diabetic foot ulcer predicts this disease

— disease descriptive, focusless

Did John Lennon meet Yoko Ono in London?

— — focusless, descriptive

Where did John Lennon meet Yoko Ono?

where where factoid, descriptive

Which drummer recorded ’Love me do’ ?

which drummer recorded

drummer descriptive, factoid

(23)

Some results

IBM’s test of Doc- tor’s Dilemma questions with Watson: Re- call@10: 0.77 (Fer- rucci et al., 2013)

FindZebra’ mean reciprocal rank on 56 questions: 0.385

(Dr˘agusin et al., 2013) IBM Watson (211 un- seen questions): Mean reciprocal rank: 0.2899, Recall@10: 0.5829.

(24)

References

Bird, S., Klein, E., and Loper, E. (2009). Natural Language Processing with Python. O’Reilly, Sebastopol, California. ISBN 9780596516499. The canonical book for the NLTK package for natural language processing in the Python programming language. Corpora, part-of-speech tagging and machine learning classification are among the topics covered.

Chu-Carroll, J., Fan, J., Schlaefer, N., and Zadrozny, W. (2012). Textual resource acquisition and engineering. IBM Journal of Research and Development, 56(3/4):4. DOI: 10.1147/JRD.2012.2185901.

Description of the unstructured corpora collected and used in the IBM Watson Jeopardy! system.

Dr˘agusin, R., Petcu, P., Lioma, C., Larsen, B., Jørgensen, H. L., Cox, I. J., Hansen, L. K., Ingwersen, P., and Winther, O. (2013). Findzebra: a search engine for rare diseases. International Journal of Medical Informatics, 82(6):528–538. DOI: 10.1016/j.ijmedinf.2013.01.005.

Ferrucci, D., Levas, A., Bagchi, S., Gondek, D., and Mueller, E. T. (2013). Watson: Beyond Jeopardy.

Artificial Intelligence, 199-200:93–105. DOI: 10.1016/j.artint.2012.06.009.

Lally, A., Prager, J. M., McCord, M. C., Boguraev, B. K., Patwardhan, S., Fan, J., Fodor, P., and Chu-Carroll, J. (2012). Question analysis: how Watson reads a clue. IBM Journal of Research and Development, 56(3.4). DOI: 10.1147/JRD.2012.2184637.