-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
321 lines (282 loc) · 24.5 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Lee Kezar</title>
<link rel="stylesheet" type="text/css" href="./styles.css">
<script src="https://code.jquery.com/jquery-3.7.1.min.js"></script>
<script src="https://code.jquery.com/ui/1.12.1/jquery-ui.min.js"></script>
<script src="app.js"></script>
</head>
<body>
<header>
<div class="container">
<span class="left-align">
<p>
<h1>Hi, I'm <span class="highlight-noexpand" style="padding-left:8px; padding-right:8px;">Lee Kezar</span></h1>
<h3>PhD Candidate, University of Southern California</h3>
I'm a Graduate Research Assistant in the <span class="highlight"><a href="http://glamor.rocks/" target="_blank">GLAMOR Lab</a></span> Lab under
<span class="highlight"><a href="https://jessethomason.com/" target="_blank">Dr. Jesse Thomason</a></span>.<br>
I'm also a Visiting Scholar in the <span class="highlight"><a href="https://sites.bu.edu/lexlab/" target="_blank">LexLab</a></span> under
<span class="highlight"><a href="http://www.naomicaselli.com/" target="_blank">Dr. Naomi Caselli</a></span> at Boston University's
<span class="highlight"><a href="https://sites.bu.edu/deafcenter/" target="_blank">Deaf Center</a></span>.
</p>
<div id="progressbar"><div class="progress-label"></div></div>
<p>✨ I'm looking for full-time roles beginning May 2025! <span class="highlight"><a href="/pdfs/LeeKezarCV.pdf" target="_blank">See my résumé</a></span></p>
</span>
<div style="flex-grow:1"><img src="images/selfie.jpeg" class="selfie"></div>
<!-- <span class="right-align">
</span> -->
</div>
</header>
<main>
<header class="container banner">
<span class="left-align">
<h2>Research Agenda</h2>
</span>
<div class="line"></div>
<span class="right-align">
<span class="expandable highlight" id="expand_concepts">Expand All</span>
<span class="expandable highlight" id="collapse_concepts">Collapse All</span>
</span>
</header>
<section style="margin-top:0;">
<p>
As language technologies like ChatGPT gain popularity, deaf people who predominantly use a sign language to communicate are increasingly left behind.
To help make these technologies more accessible, I use <span class="super expandable concept highlight" id="ling">sign language linguistics</span> to make models that can <span class="super expandable concept highlight" id="understanding">understand sign language</span>.
My current work focuses on <span class="expandable concept highlight" id="semantics">lexical semantics</span>, leveraging <span class="expandable concept highlight" id="knowledgebase">linguistic knowledge bases</span> to improve model
<span class="expandable concept highlight" id="perception">perception</span>, <span class="expandable concept highlight" id="recognition">recognition</span>, and <span class="expandable concept highlight" id="comprehension">comprehension</span>.
</p>
</section>
<section class="super concept expansion" id="understanding">
<p>
<span class="highlight-noexpand">Understanding Sign Language</span><br>
Understanding any sign language requires <span class="expandable concept highlight" id="perception">perceiving</span> visual features like the location and configuration of the hands, <span class="expandable concept highlight" id="recognition">recognizing</span> what those features signify, and
<span class="expandable concept highlight" id="comprehension">comprehending</span> the intended meaning through inference.
To satisfy these requirements, I adopt a <span class="super expandable concept highlight" id="ling">linguistic</span> approach.
My work presents the argument that a model which can recognize phonological features not only recognizes isolated signs more accurately, but also enables it to approximate the meaning of signs it has never seen before!
</p>
<section class="concept expansion" id="perception">
<p>
<span class="highlight-noexpand">Perception</span><br>
Understanding sign language begins with perceiving many subtle visual features called <span class="expandable concept highlight" id="phonology">phonemes</span>, for example the flexion of each finger or the path that the hands move along.
Our EACL paper <span class="expandable paper highlight" id="eacl">Improving Sign Recognition with Phonology</span> shows that models miss many of these features, leading to poorer performance in higher-level tasks like recognizing signs.
To improve models' phonological perception, a follow-up work
<span class="expandable paper highlight" id="esann">Exploring Strategies for Modeling Sign Language Phonology</span> provides a method which recognizes 16 types of phonological features with 75%-90% accuracy using a technique called curriculum learning.
</p>
</section>
<section class="concept expansion" id="recognition">
<p>
<span class="highlight-noexpand">Recognition</span><br>
Recognizing which sign(s) are present in a video is a critical step towards automatic sign language translation and requires distinguishing many visually-similar signs.
This is a difficult task for rare and modified signs, which are unlikely to have been seen in training.
Our paper, <span class="expandable paper highlight" id="eacl">Improving Sign Recognition with Phonology</span> showed that models are better at distinguishing signs when they can also recognize the pieces of those signs, i.e. <span class="expandable concept highlight" id="phonology">phonemes</span>.
This result was replicated in <span class="expandable paper highlight" id="assets">The Sem-Lex Benchmark: Modeling ASL Signs and Their Phonemes</span>, adding that rare signs benefit the most from phoneme recognition.
</p>
</section>
<section class="concept expansion" id="comprehension">
<p>
<span class="highlight-noexpand">Comprehension</span><br>
One powerful benefit of modeling sign language <span class="expandable concept highlight" id="phonology">phonology</span> is the ability to apply inference to the phonemes after they are recognized.
Specifically, by leveraging systematic relationships between form and meaning, we may be able to infer the meaning of a sign even if the model has never seen it before!
Ongoing work explores this possibility, which we intend to publish in 2024.
</p>
</section>
</section>
<section class="super concept expansion" id="ling">
<p>
<span class="highlight-noexpand">Sign Language Linguistics</span><br>
Sign languages are like spoken languages in almost every way.
Each sign language has its own vocabulary and grammar, and they are independent of any spoken language used in their region.
They have a complex structure at all levels, such as <span class="expandable concept highlight" id="phonology">phonology</span> and <span class="expandable concept highlight" id="semantics">semantics</span>.
My work leverages the relationship between phonology and semantics, also known as systematicity, to help machine learning models understand signs they haven't seen before. <span class="super expandable concept highlight" id="understanding">understand sign language</span>
</p>
<section class="concept expansion" id="phonology">
<p>
<span class="highlight-noexpand">Phonemes</span><br>
Phonemes are the smallest units of language.
In sign languages, phonemes fall into one of several categories, such as hand configuration, movement, and location.
My work loosely falls under Brentari's Prosodic Model, which emphasizes the simultaneity and interdependency of phonemes.
</p>
</section>
<section class="concept expansion" id="semantics">
<p>
<span class="highlight-noexpand">Lexical Semantics</span><br>
Many times, the meaning of a sign has a direct relationship to its form.
This relationship, called systematicity, provides valuable scaffolding to learn signs more efficiently and can enable understanding even when the sign has never been seen before.
Currently, my team is working on a large knowledge graph of ASL signs' form and meaning, which we will be releasing in Fall 2024.
We hope this resource will be used in creative ways to computationally model ASL systematicity.
</p>
</section>
<section class="concept expansion" id="knowledgebase">
<p>
<span class="highlight-noexpand">Linguistic Knowledge Bases</span><br>
Structured, accurate knowledge about the structure of signs can be a powerful addition to current models for sign language.
For example, training models to simultaneously recognize phonological features alongside sign identifiers improves their accuracy at both tasks.
My work relies on knowledge bases like <span class="highlight"><a href="https://asl-lex.org/" target="_blank">ASL-LEX 2.0</a></span>, a dataset of linguistic and cognitive facts related to signs, and <span class="highlight"><a href="https://hci.stanford.edu/publications/2016/ethan/empath-chi-2016.pdf" target="_blank">Empath</a></span>, a dataset of semantic categories for many English words.
We've also released a dataset of our own, <span class="expandable paper highlight" id="assets">The Sem-Lex Benchmark</span>, containing over 91,000 isolated sign videos representing a vocabulary of 3,100 signs (the largest of its kind!).
</p>
</section>
</section>
<header class="container banner">
<span class="left-align"><h2>Publications</h2></span>
<div class="line"></div>
<span class="right-align">
<span class="expandable highlight" id="expand_papers">Expand All</span>
<span class="expandable highlight" id="collapse_papers">Collapse All</span>
</span>
</header>
<h3>2023</h3>
<section style="margin-top:0">
<b>Lee Kezar</b>, Elana Pontecorvo, Adele Daniels, Connor Baer, Ruth Ferster, Lauren Berger, Jesse Thomason, Zed Sehyr, and Naomi Caselli.
<span class="expandable highlight paper" id="assets" style="margin-bottom:0">The Sem-Lex Benchmark: Modeling ASL Signs and Their Phonemes</span>. Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2023).
</section>
<section class="paper expansion" id="assets">
<div class="container">
<span class="left-align">
<p>
<span class="highlight-noexpand">TLDR</span><br>
The Sem-Lex Benchmark provides 91,000 isolated sign videos for the tasks of recognizing signs and 16 phonological feature types.
</p>
<p>
<span class="highlight-noexpand">Abstract</span><br>
Sign language <span class="expandable concept highlight" id="recognition">recognition</span> and translation technologies have the potential to increase access and inclusion of deaf signing communities, but research progress is bottlenecked by a lack of representative data. We introduce a new resource for American Sign Language (ASL) modeling, the Sem-Lex Benchmark. The Benchmark is the current largest of its kind, consisting of over 84k videos of isolated sign productions from deaf ASL signers who gave informed consent and received compensation. Human experts aligned these videos with other sign language resources including ASL-LEX, SignBank, and ASL Citizen, enabling useful expansions for sign and <span class="expandable concept highlight" id="perception">phonological feature recognition</span>. We present a suite of experiments which make use of the linguistic information in ASL-LEX, evaluating the practicality and fairness of the Sem-Lex Benchmark for isolated sign recognition (ISR). We use an SL-GCN model to show that the phonological features are recognizable with 85% accuracy, and that they are efective as an auxiliary target to ISR. Learning to recognize phonological features alongside gloss results in a 6% improvement for few-shot ISR accuracy and a 2% improvement for ISR accuracy overall. Instructions for downloading the data can be found at https://github.com/leekezar/SemLex.
</p>
</span>
<div style="flex-grow: 1"></div>
<span class="right-align" style="padding:0 5% 0 5%; text-align:center;">
<i>Click to view paper</i><br>
<a href="https://dl.acm.org/doi/pdf/10.1145/3597638.3608408" target="_blank ">
<img src="./images/ASSETS.png" style="height:200px;">
</a>
<br>
<span class="highlight">
<a href="https://docs.google.com/presentation/d/1eilvWClHUFkbILgu3zuVmPEqs16N4ktYglVQN4E1XJY/edit?usp=sharing" target="_blank">Slides</a>
</span>
<span class="highlight">
<a href="./bibtex/ASSETS.bib.bib" download="Kezar2023SemLex">Download BibTeX</a>
</span>
<span class="highlight">
<a href="https://docs.google.com/forms/d/e/1FAIpQLSeFjIcbJcr2kWibgrEdFyLhNADo1ErnVGuQHtGeiDiqe4iteQ/viewform?usp=sf_link" target="_blank">Download Sem-Lex</a>
</span>
</span>
</div>
</section>
<section style="margin-top:0">
<b>Lee Kezar</b>, Riley Carlin, Tejas Srinivasan, Zed Sehyr, Naomi Caselli, and Jesse Thomason.
<span class="expandable paper highlight" id="esann" style="margin-bottom:0">Exploring Strategies for Modeling Sign Language Phonology</span>. Proceedings of the 31st European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN 2023).
</section>
<section class="paper expansion" id="esann">
<div class="container">
<span class="left-align">
<p>
<span class="highlight-noexpand">TLDR</span><br>
We train SL-GCN models to recognize 16 phonological feature types (like handshape and location), achieving 75-90% accuracy.
</p>
<p>
<span class="highlight-noexpand">Abstract</span><br>
Like speech, signs are composed of discrete, recombinable features called <span class="expandable concept highlight" id="phonology">phonemes</span>. Prior work shows that models which can <span class="expandable concept highlight" id="perception">recognize phonemes</span> are better at <span class="expandable concept highlight" id="recognition">sign recognition</span>, motivating deeper exploration into strategies for modeling sign language phonemes. In this work, we learn graph convolution networks to recognize the sixteen phoneme"types"found in ASL-LEX 2.0. Specifically, we explore how learning strategies like multi-task and curriculum learning can leverage mutually useful information between phoneme types to facilitate better modeling of sign language phonemes. Results on the Sem-Lex Benchmark show that curriculum learning yields an average accuracy of 87% across all phoneme types, outperforming fine-tuning and multi-task strategies for most phoneme types.
</p>
</span>
<div style="flex-grow: 1"></div>
<span class="right-align" style="padding:0 5% 0 5%; text-align:center;">
<i>Click to view paper</i><br>
<a href="https://www.esann.org/sites/default/files/proceedings/2023/ES2023-83.pdf" target="_blank ">
<img src="./images/ESANN.png" style="height:200px;">
</a>
<br>
<span class="highlight">
<a href="https://docs.google.com/presentation/d/1Eym6Qxn3ErXUx9xR16ddx2jVUvS8Xu_zPCIJ7gwTzGQ/edit?usp=sharing" target="_blank">Slides</a>
</span>
<span class="highlight">
<a href="./bibtex/ESANN.bib" download="Kezar2023ESANN.bib">Download BibTeX</a>
</span>
<!-- -->
<!-- <span class="highlight">
<a href="https://docs.google.com/forms/d/e/1FAIpQLSeFjIcbJcr2kWibgrEdFyLhNADo1ErnVGuQHtGeiDiqe4iteQ/viewform?usp=sf_link" target="_blank">Download Sem-Lex</a>
</span> -->
</span>
</div>
</section>
<section style="margin-top:0">
<b>Lee Kezar</b>, Jesse Thomason, and Zed Sehyr.
<span class="expandable highlight paper" id="eacl" style="margin-bottom:0">Improving Sign Recognition with Phonology</span>. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2023).
</section>
<section class="paper expansion" id="eacl">
<div class="container">
<span class="left-align">
<p>
<span class="highlight-noexpand">TLDR</span><br>
We show that adding phonological targets boosts sign recognition accuracy by ~9%!
</p>
<p>
<span class="highlight-noexpand">Abstract</span><br>
We use insights from research on American Sign Language (ASL) <span class="expandable concept highlight" id="phonology">phonology</span> to train models for <span class="expandable concept highlight" id="recognition">isolated sign language recognition</span> (ISLR), a step towards automatic sign language <span class="expandable concept highlight" id="understanding">understanding</span>. Our key insight is to explicitly recognize the role of phonology in sign production to achieve more accurate ISLR than existing work which does not consider sign language phonology. We train ISLR models that take in pose estimations of a signer producing a single sign to predict not only the sign but additionally its phonological characteristics, such as the handshape. These auxiliary predictions lead to a nearly 9% absolute gain in sign recognition accuracy on the WLASL benchmark, with consistent improvements in ISLR regardless of the underlying prediction model architecture. This work has the potential to accelerate linguistic research in the domain of signed languages and reduce communication barriers between deaf and hearing people.
</p>
</span>
<div style="flex-grow: 1"></div>
<span class="right-align" style="padding:0 5% 0 5%; text-align:center;">
<i>Click to view paper</i><br>
<a href="https://aclanthology.org/2023.eacl-main.200.pdf" target="_blank ">
<img src="./images/EACL.png" style="height:200px;">
</a>
<br>
<span class="highlight">
<a href="https://docs.google.com/presentation/d/1XFD0LWofX0uZvUJDLIOvGbg7LiFT6I1xpJeQgw7NOJk/edit?usp=sharing" target="_blank">Slides</a>
</span>
<span class="highlight">
<a href="https://docs.google.com/presentation/d/1cT_EqWUga1Rxuf_aNtCWhTy3xYodERGebGzW_Npzqto/edit?usp=sharing" target="_blank">Poster</a>
</span>
<span class="highlight">
<a href="./bibtex/EACL.bib" download="Kezar2023EACL.bib">Download BibTeX</a>
</span>
<!-- -->
<!-- <span class="highlight">
<a href="https://docs.google.com/forms/d/e/1FAIpQLSeFjIcbJcr2kWibgrEdFyLhNADo1ErnVGuQHtGeiDiqe4iteQ/viewform?usp=sf_link" target="_blank">Download Sem-Lex</a>
</span> -->
</span>
</div>
</section>
<h3>2021</h3>
<section style="margin-top:0">
<b>Lee Kezar</b> and Jay Pujara.
<span class="expandable highlight paper" id="sdp" style="margin-bottom:0">Finding Pragmatic Differences Between Disciplines</span>. Proceedings of the Second Workshop on Scholarly Document Processing (SDP 2021).
</section>
<section class="expansion paper" id="sdp">
<div class="container">
<span class="left-align">
<p>
<span class="highlight-noexpand">TLDR</span><br>
We classify sections in scholarly documents according to their pragmatic intent and study the differences between 19 disciplines.
</p>
<p>
<span class="highlight-noexpand">Abstract</span><br>
Scholarly documents have a great degree of variation, both in terms of content (semantics) and structure (pragmatics). Prior work in scholarly document understanding emphasizes semantics through document summarization and corpus topic modeling but tends to omit pragmatics such as document organization and flow. Using a corpus of scholarly documents across 19 disciplines and state-of-the-art language modeling techniques, we learn a fixed set of domain-agnostic descriptors for document sections and “retrofit” the corpus to these descriptors (also referred to as “normalization”). Then, we analyze the position and ordering of these descriptors across documents to understand the relationship between discipline and structure. We report within-discipline structural archetypes, variability, and between-discipline comparisons, supporting the hypothesis that scholarly communities, despite their size, diversity, and breadth, share similar avenues for expressing their work. Our findings lay the foundation for future work in assessing research quality, domain style transfer, and further pragmatic analysis.
</p>
</span>
<div style="flex-grow: 1"></div>
<span class="right-align" style="padding:0 5% 0 5%; text-align:center;">
<i>Click to view paper</i><br>
<a href="https://aclanthology.org/2021.sdp-1.10.pdf" target="_blank ">
<img src="./images/SDP.png" style="height:200px;">
</a>
<br>
<span class="highlight">
<a href="./bibtex/SDP.bib" download="Kezar2021ACL.bib">Download BibTeX</a>
</span>
<!-- -->
<!-- <span class="highlight">
<a href="https://docs.google.com/forms/d/e/1FAIpQLSeFjIcbJcr2kWibgrEdFyLhNADo1ErnVGuQHtGeiDiqe4iteQ/viewform?usp=sf_link" target="_blank">Download Sem-Lex</a>
</span> -->
</span>
</div>
</section>
</main>
</body>
</html>