BEGIN:VCALENDAR
VERSION:2.0
METHOD:PUBLISH
CALSCALE:GREGORIAN
PRODID:-//WordPress - MECv6.5.6//EN
X-ORIGINAL-URL:https://cmc.deusto.eus/
X-WR-CALNAME:cmc.deusto.eus
X-WR-CALDESC:DeustoCCM - Chair of Computational Mathematics at University of Deusto
REFRESH-INTERVAL;VALUE=DURATION:PT1H
X-PUBLISHED-TTL:PT1H
X-MS-OLK-FORCEINSPECTOROPEN:TRUE
BEGIN:VEVENT
CLASS:PUBLIC
UID:MEC-807c4063f8fd3da43661d3087d697c7e@cmc.deusto.eus
DTSTART:20240502T100000Z
DTEND:20240502T110000Z
DTSTAMP:20251031T212700Z
CREATED:20251031
LAST-MODIFIED:20251031
PRIORITY:5
TRANSP:OPAQUE
SUMMARY:Clustering in Pure-Attention Hardmax Transformers
DESCRIPTION:Next Thursday May 2, 2024:\nOrganized by: FAU DCN-AvH, Chair for Dynamics, Control, Machine Learning and Numerics – Alexander von Humboldt Professorship at FAU, Friedrich-Alexander-Universität Erlangen-Nürnberg (Germany)\nTitle: Clustering in Pure-Attention Hardmax Transformers\nSpeaker: Albert Alcalde\nAffiliation: PhD student at FAU DCN-AvH Chair for Dynamics, Control, Machine Learning and Numerics – Alexander von Humboldt Professorship.\nAbstract.  We study the behaviour in the infinite-depth limit of a transformer model with hardmax self-attention and normalization sublayers, by viewing it as a discrete-time dynamical system acting on a collection of points. Leveraging a simple geometric interpretation of our transformer connected with ideas of hyperplane separation, we establish convergence to a clustered equilibrium and prove that clusters are completely determined by special points called leaders. We apply our theoretical understanding to design a model based on our transformer to solve the sentiment analysis task in an interpretable way: the transformer filters out meaningless words by clustering them towards the leaders, identified with words carrying the sentiment of the text such as ‘amazing’ or ‘terrible’.\nWHEN\nThu. May 2, 2024 at 12:00H\nWHERE\nOn-site: Room 03.323\nFriedrich-Alexander-Universität Erlangen-Nürnberg\nCauerstraße 11, 91058 Erlangen\nGPS-Koord. Raum: 49.573764N, 11.030028E\n_\nSee all Seminars at FAU DCN-AvH\nDon’t miss out our last news and connect with us!\nLinkedIn | Twitter | Instagram\n
URL:https://cmc.deusto.eus/events-calendar/clustering-in-pure-attention-hardmax-transformers/
ORGANIZER;CN=FAU DCN-AvH:MAILTO:
CATEGORIES:FAU DCN-AvH Jr. Seminar
ATTACH;FMTTYPE=image/png:https://cmc.deusto.eus/wp-content/uploads/FAUDCNAvHJrSeminar_aAlcalde_02may2024.png
END:VEVENT
END:VCALENDAR