<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:media="http://search.yahoo.com/mrss/"><channel><title><![CDATA[Nelson Liu's Blog]]></title><description><![CDATA[Nelson Liu's Blog]]></description><link>https://blog.nelsonliu.me/</link><image><url>https://blog.nelsonliu.me/favicon.png</url><title>Nelson Liu&apos;s Blog</title><link>https://blog.nelsonliu.me/</link></image><generator>Ghost 4.48</generator><lastBuildDate>Wed, 22 Apr 2026 11:07:44 GMT</lastBuildDate><atom:link href="https://blog.nelsonliu.me/rss/" rel="self" type="application/rss+xml"/><ttl>60</ttl><item><title><![CDATA[PhD Statement of Purpose]]></title><description><![CDATA[<p>I&apos;ve been pretty liberal about sharing my PhD statement of purpose with folks who email me, so in the interest of open and equal access, I&apos;m putting them here for anyone to see. I also included some notes on thoughts I have looking back, with some</p>]]></description><link>https://blog.nelsonliu.me/2020/11/11/phd-personal-statement/</link><guid isPermaLink="false">5fab98cee0614f64c32bcb56</guid><dc:creator><![CDATA[Nelson Liu]]></dc:creator><pubDate>Wed, 11 Nov 2020 08:25:27 GMT</pubDate><content:encoded><![CDATA[<p>I&apos;ve been pretty liberal about sharing my PhD statement of purpose with folks who email me, so in the interest of open and equal access, I&apos;m putting them here for anyone to see. I also included some notes on thoughts I have looking back, with some things I would have changed if I were to rewrite it. If you&apos;re looking for advice on writing statements of purpose, I like <a href="https://docs.google.com/document/d/1lT-bsIP0GKfh8l5sQnM2hCzzR9prt-QLx16rimUOdIM/edit">this document</a> by Noah Smith (much of this advice shaped the final document below) and <a href="https://nschneid.medium.com/inside-ph-d-admissions-what-readers-look-for-in-a-statement-of-purpose-3db4e6081f80">this document</a> by Nathan Schneider. I applied in December 2018 [1]:</p><p><a href="https://nelsonliu.me/files/phd_sop/nfliu_phd_sop.pdf">PhD Statement of Purpose</a></p><p>I hesitate to do this because it&apos;s pretty cringe in hindsight (and my interests have definitely shifted considerably since writing this), but I think it does a reasonable job of avoiding a common failure mode: making your statement of purpose just your CV, but translated into paragraphs. I wrote a fair amount about the past projects that I worked on, but I mainly did so towards the goal of motivating / painting a coherent picture of how I reached my research interests at the time.</p><p>For example, the paragraph about my past work at AI2 was mostly to convey that I had hands-on experience with QA models and was pretty disillusioned by the gap between their (hyped) purported capabilities and their actual capabilities.</p><p>I also think the paragraph about future career goals could be omitted&#x2014;don&apos;t feel pressured to make any semblance of a decision of what you want to do post-PhD. I thought it might just be useful to mention, since I really enjoyed TAing as an undergrad and one of my letter-writers was someone who I had pretty much only interacted with in the context of TA-ing. You will not be held accountable for whatever career goals you set, people expect them to change :) (and I can confirm that your opinions will certainly develop during your PhD).</p><p>If I could change things, I&apos;d definitely write less about my past. For instance, I would shorten the part about the RepL4NLP paper&#x2014;there&apos;s no need to go into the concrete results, it would suffice to just say that my personal experience with the opacity of neural nets led me to do some initial work in the area, and motivated my future interests.</p><p>I also regret not being more concrete about maybe specific projects I want to do in the future, but I recognize that it&apos;s easy to say that in hindsight. As I was writing the statement, I was definitely afraid that my misinformed senior-year undergraduate research opinions would turn off any NLP faculty with the misfortune of reading my application, so I chose to be conservative instead and say less. Looking back, I think it might have actually looked better to have stronger personal opinions with more evidence for why I feel the way I do.</p><p>Have confidence in your ideas and research goals; you&apos;ve clearly thought them through, so use the statement to convey why <em>you</em> think these problems are interesting, and why your proposed solutions seem reasonable in light of prior work in the area.</p><p>[1]: For historical context, BERT was released in October 2018, and BERTology wasn&apos;t really a thing yet&#x2014;anyone writing about analyzing neural nets today probably needs to write more arguing why this is interesting, and what new things they want to bring to the table.</p>]]></content:encoded></item><item><title><![CDATA[Newer PyTorch Binaries for Older GPUs]]></title><description><![CDATA[<p>After recently upgrading some code to a newer version of PyTorch, I found that it would no longer successfully execute on NVIDIA Tesla K40 GPUs. In particular, I was seeing the following error whenever a model&apos;s <code>.forward()</code> function was called:</p><pre><code>RuntimeError: CUDA error: no kernel image is available</code></pre>]]></description><link>https://blog.nelsonliu.me/2020/10/13/newer-pytorch-binaries-for-older-gpus/</link><guid isPermaLink="false">5f1e405d1d23010b77dcb210</guid><dc:creator><![CDATA[Nelson Liu]]></dc:creator><pubDate>Tue, 13 Oct 2020 21:54:03 GMT</pubDate><content:encoded><![CDATA[<p>After recently upgrading some code to a newer version of PyTorch, I found that it would no longer successfully execute on NVIDIA Tesla K40 GPUs. In particular, I was seeing the following error whenever a model&apos;s <code>.forward()</code> function was called:</p><pre><code>RuntimeError: CUDA error: no kernel image is available for execution on the device</code></pre><p>Turns out that PyTorch v1.3.1 <a href="https://github.com/pytorch/pytorch/issues/30532#issuecomment-559258091">dropped support for GPUs with NVIDIA compute capability 3.5</a> in their prebuilt binaries that you&apos;d get from <code>pip</code> or <code>conda</code> &#x2014;the stated reason was that supporting these older GPUs would have pushed binary sizes past acceptable limits for distribution.</p><p>I built new binaries that add back support for these older GPUs (e.g., K40) and will be uploading them at <a href="https://github.com/nelson-liu/pytorch-manylinux-binaries/releases">https://github.com/nelson-liu/pytorch-manylinux-binaries/releases</a> for each PyTorch release. You can also find the download links at <a href="https://nelsonliu.me/files/pytorch/whl/torch_stable.html" rel="nofollow">nelsonliu.me/files/pytorch/whl/torch_stable.html</a> , and these binaries are pip-installable via (change the desired PyTorch / CUDA version, as necessary):</p><pre><code>pip install torch==1.3.1+cu92 -f https://nelsonliu.me/files/pytorch/whl/torch_stable.html</code></pre>]]></content:encoded></item><item><title><![CDATA[NSF GRFP Application Materials]]></title><description><![CDATA[<p>I&apos;ve been pretty liberal about sharing my NSF GRFP materials with folks who email me, so in the interest of open and equal access, I&apos;m putting them here for anyone to see. I applied for the<em> </em>fellowship in late-2018 as a final-year undergraduate, and was a</p>]]></description><link>https://blog.nelsonliu.me/2020/10/13/nsf-grfp-materials/</link><guid isPermaLink="false">5f861cb1e0614f64c32bcb12</guid><dc:creator><![CDATA[Nelson Liu]]></dc:creator><pubDate>Tue, 13 Oct 2020 21:43:26 GMT</pubDate><content:encoded><![CDATA[<p>I&apos;ve been pretty liberal about sharing my NSF GRFP materials with folks who email me, so in the interest of open and equal access, I&apos;m putting them here for anyone to see. I applied for the<em> </em>fellowship in late-2018 as a final-year undergraduate, and was a fortunate recipient.</p><p><a href="http://nelsonliu.me/files/nsf_grfp/nfliu_nsf_grfp_research_statement.pdf">Research Statement</a></p><p><a href="http://nelsonliu.me/files/nsf_grfp/nfliu_nsf_grfp_reviews.pdf">Reviews</a></p><p>I was a bit personal in my personal statement, so I&apos;m not planning on sharing it publicly. At a high level, the structure looked like:</p><ul><li>Background / my motivations for pursuing a research career</li><li>The various past projects I&apos;ve worked on, with concrete examples of what I learned from each of them and how they&apos;ve helped me develop as a researcher. For instance, &quot;Designing this model was a highly iterative process, and took over a year&#x2014;struggling through numerous prototypes gave me experience with interpreting negative results to improve my method.&quot;</li><li>A section on broader impacts, mostly focusing on my participation in outreach activities and my involvement in open-source software development.</li><li>A brief paragraph on future goals.</li></ul><p>Good luck with your application!</p>]]></content:encoded></item><item><title><![CDATA[Fixing system permissions when writing to Docker volumes]]></title><description><![CDATA[<p>I&apos;ve been using Docker a lot recently, it&apos;s a great way to run old code (think 2016-era Theano code) and ensure reproducible setups across machines. I typically mount my source code as a Docker volume, so I read and write to the directory from my container.</p>]]></description><link>https://blog.nelsonliu.me/2020/03/28/fixing-permissions/</link><guid isPermaLink="false">5e7e9ca22aa85523bca5b6de</guid><dc:creator><![CDATA[Nelson Liu]]></dc:creator><pubDate>Sat, 28 Mar 2020 00:49:18 GMT</pubDate><content:encoded><![CDATA[<p>I&apos;ve been using Docker a lot recently, it&apos;s a great way to run old code (think 2016-era Theano code) and ensure reproducible setups across machines. I typically mount my source code as a Docker volume, so I read and write to the directory from my container. </p><p></p><p>However, because Docker containers are privileged, the output files are owned by <code>root</code> , and I can&apos;t even delete them once I return to my system. A quick fix is to launch another Docker container that simply <code>chown</code>s everything in the current directory, recursively:</p><pre><code>docker run -it --rm \
    -v $(pwd):/workdir \
    --workdir /workdir \
    alpine \
    chown -R $(id -u):$(id -g) .</code></pre>]]></content:encoded></item><item><title><![CDATA[Student Perspectives on Applying to NLP PhD Programs]]></title><description><![CDATA[This post offers advice and perspectives on the NLP PhD application process. We asked 12 recent applicants a range of questions about their experience, and summarized the similarities and differences in the shared experience.]]></description><link>https://blog.nelsonliu.me/2019/10/24/student-perspectives-on-applying-to-nlp-phd-programs/</link><guid isPermaLink="false">5dafee482aa85523bca5b597</guid><category><![CDATA[nlp]]></category><category><![CDATA[phd]]></category><category><![CDATA[research]]></category><category><![CDATA[applications]]></category><category><![CDATA[advice]]></category><dc:creator><![CDATA[Nelson Liu]]></dc:creator><pubDate>Thu, 24 Oct 2019 17:23:03 GMT</pubDate><media:content url="https://blog.nelsonliu.me/content/images/2019/10/thumbnail.gif" medium="image"/><content:encoded><![CDATA[<!--kg-card-begin: markdown--><img src="https://blog.nelsonliu.me/content/images/2019/10/thumbnail.gif" alt="Student Perspectives on Applying to NLP PhD Programs"><p><strong>This post was written by</strong>: <a href="https://akariasai.github.io/">Akari Asai</a>, <a href="https://nlp.stanford.edu/~johnhew/">John Hewitt</a>, <a href="https://www.siddkaramcheti.com/">Sidd Karamcheti</a>, <a href="http://martiansideofthemoon.github.io/">Kalpesh Krishna</a>, <a href="http://nelsonliu.me">Nelson Liu</a>, <a href="http://cs.brown.edu/people/rpatel59/">Roma Patel</a>, and <a href="https://people.eecs.berkeley.edu/~nicholas_tomlin/">Nicholas Tomlin</a>.</p>
<p><strong>Thanks to our amazing survey respondents</strong>: <a href="https://akariasai.github.io/">Akari Asai</a>, <a href="https://ashkamath.github.io/">Aishwarya Kamath</a>,  <a href="https://www.siddkaramcheti.com/">Sidd Karamcheti</a>, <a href="http://martiansideofthemoon.github.io/">Kalpesh Krishna</a>, <a href="https://lucy3.github.io/">Lucy Li</a>, <a href="https://people.eecs.berkeley.edu/~kevinlin/">Kevin Lin</a>, <a href="http://nelsonliu.me">Nelson Liu</a>, <a href="https://sjmielke.com/">Sabrina Mielke</a>, <a href="http://cs.brown.edu/people/rpatel59/">Roma Patel</a>, <a href="https://people.eecs.berkeley.edu/~nicholas_tomlin/">Nicholas Tomlin</a>, <a href="http://www.ericswallace.com/">Eric Wallace</a>, and <a href="https://cs.stanford.edu/~myasu/">Michihiro Yasunaga</a>.</p>
<p>This post offers and summarizes student advice and perspectives on the NLP PhD application process, with a focus on programs in the US. We asked twelve recently-successful NLP PhD applicants a range of questions about the application process&#x2014;this post compiles the broader themes and advice that run through the majority of responses. Make sure to check out <a href="https://drive.google.com/open?id=1hOSV633f59IKAdKNB6attME6O8qAIUB1"><strong>the complete set of responses</strong></a>! <a href="http://nelsonliu.me/files/nlp_phd_application_survey_responses.tar">A tarball is also available for those who cannot access Google Drive</a>.</p>
<p>&#x26A0;&#xFE0F;<strong>Disclaimer</strong>&#x26A0;&#xFE0F;: While we&#x2019;ve all gone through the application process and have thoughts to share, we aren&#x2019;t experts or authorities on this (highly random) process. Our advice comes from our unique perspectives and backgrounds, and not everything will generalize. That said, we hope that the differences and similarities in our shared experiences will be useful to consider.</p>
<p>Professors have also written advice to applicants from their side of the process, see <a href="http://martiansideofthemoon.github.io/2018/05/29/grad-resources.html">Kalpesh Krishna&#x2019;s compilation of graduate school application advice</a>.</p>
<h1 id="tableofcontentsanametableofcontentshreftableofcontentsiclassfafalinkia">Table of Contents <a name="table-of-contents" href="#table-of-contents"><i class="fa fa-link"></i></a></h1>
<ul>
<li><a href="#pre-application">Pre-application</a></li>
<li><a href="#statement-of-purpose">Statement of Purpose</a></li>
<li><a href="#letters-of-recommendation">Letters of Recommendation</a></li>
<li><a href="#publications">Publications</a></li>
<li><a href="#transcripts--grades">Transcripts / Grades</a></li>
<li><a href="#standardized-exams-gre--toefl">Standardized Exams: GRE / TOEFL</a></li>
<li><a href="#interviews--post-application-calls">Interviews / Post-application Calls</a></li>
<li><a href="#deciding-where-to-go">Deciding where to go</a></li>
<li><a href="#misc-topics">Misc. Topics</a></li>
<li><a href="#in-conclusion">In Conclusion</a></li>
</ul>
<hr>
<h1 id="preapplicationanamepreapplicationhrefpreapplicationiclassfafalinkia">Pre-application <a name="pre-application" href="#pre-application"><i class="fa fa-link"></i></a></h1>
<p>Deciding to apply at all is not an easy choice, and several respondents took additional time, either in school or in industry, to explore new fields and become more certain that pursuing a PhD was the right decision for them. Choosing where to apply is also an involved process, and involves trade-offs between factors like research area fit, location, and (perceived) selectivity. This section explores this preliminary part of the application process, along with useful insights from applicants on different aspects of this decision.</p>
<p>A lot of the perspectives in this post are aimed towards people already seriously considering a PhD&#x2014;for instance, seniors or MS students. If you are a student considering a PhD, but still have a significant amount of time before you apply, <a href="https://nlp.stanford.edu//~johnhew//undergrad-researchers.html">John Hewitt&#x2019;s blog post</a> contains useful insights and advice on how to make the most of your time in school. In addition, <a href="http://martiansideofthemoon.github.io/2018/05/29/grad-resources.html">Kalpesh Krishna&#x2019;s extensive compilation of application advice </a> might yield things to keep in mind through the years.</p>
<h2 id="whyapplynow">Why apply now?</h2>
<p>For many of the respondents, starting a PhD was the natural &#x201C;next step&#x201D;&#x2014;they were in the final year of their undergraduate or masters degrees, and had spent enough time doing research to realize that a PhD was worth the opportunity cost to them.</p>
<blockquote>
<p>While I did not have any *ACL papers while applying...My goal was to get into a good PhD program and start doing research full-time (which is why I was applying to a PhD program in the first place) rather than get into the very best PhD program.<br>
&#x2013; Kalpesh Krishna</p>
</blockquote>
<p>Waiting to apply also has clear benefits&#x2014;many respondents felt that they would be stronger applicants after an additional year of research experience (and the associated publications and stronger letters of recommendation that might come with it).</p>
<blockquote>
<p>&#x201C;The year away from academia gave me the clarity on how much I really wanted to do a PhD and how much I love academic life. In this year I used my free time to explore interesting research directions and collaborated with friends. It made me realise that I enjoy research and to be able to do it for a living would be just perfect.&#x201D;<br>
&#x2013; Aishwarya Kamath</p>
</blockquote>
<blockquote>
<p>&#x201C;I was also unsure at that time what kinds of directions I wanted to go in or if I even wanted to commit so many years of my life to additional school...By the time fall 2018 came around, I&#x2019;d done a full year of thinking and growing my research skills, so I felt a lot better about diving into the process.&#x201D;<br>
&#x2013; Lucy Li</p>
</blockquote>
<p>Several people found value in waiting because it gave them the time to reflect on their next steps. For instance, Lucy and Aishwarya used the time to further develop their research interests and think about what areas were exciting to them. In particular, Aishwarya spent a year in industry, which made her realize what she was missing an academic setting and drove her to apply and return.</p>
<p>On the other hand, several also offered caution about waiting with the sole intention of improving your profile. As PhD applications get more and more competitive each year,  more papers or experience doesn&#x2019;t necessarily mean a stronger application since things are inherently relative. Several agreed that having publications at top conferences is not a necessary component of a strong application, especially if one has relatively limited research experience (e.g., applicants from undergrad) or has strong recommendation letters. <a href="https://timdettmers.com/2018/11/26/phd-applications/">A recent blog post about the machine learning PhD application process</a> investigates admission statistics at one of the top schools (Fall 2018), and shows admission is not determined solely on publication records, but depends on the other factors, especially applicants&#x2019; background and letters of recommendations.</p>
<p>For instance, Kalpesh and Akari considered waiting a year since they did not have any top-tier NLP publications at the time, noting that:</p>
<ul>
<li>Things get more and more competitive each year, so more papers doesn&#x2019;t necessarily mean a stronger application since things are inherently relative.</li>
<li>Applicants with master&apos;s degrees are expected to have more publications and experience than undergraduate applicants.</li>
<li>There is a large amount of uncertainty involved in research / writing papers, so things are not always going to pan out for reasons out of your control.</li>
<li>They thought that they were still reasonably strong applicants for many of the places they were applying to.</li>
</ul>
<p>Kevin and Akari also mention that, if you have the resources, you can apply multiple times.</p>
<blockquote>
<p>If what you really want to do is to immediately get into a grad school and continue doing work that you are excited about, you should apply.<br>
&#x2013; Roma Patel</p>
</blockquote>
<h2 id="choosingwheretoapply">Choosing where to apply</h2>
<p>When choosing where to apply, the majority of respondents focused on a few factors:</p>
<ol>
<li>Overwhelmingly, the strongest factor for everyone was <em><strong>faculty</strong></em>: finding schools with professors that you&#x2019;d want to work with, and with a strong presence in allied fields. Several mentioned applying to places only if there were 2 or more relevant faculty.</li>
<li><em><strong>Location</strong></em> was also a key factor for many: finding schools in places that you think you&#x2019;d be happy living in for 5+ years.</li>
<li>Lastly, many also considered <em><strong>proximity to industry connections / possible external collaborators</strong></em>.</li>
</ol>
<p>Some also took the relative prestige of a school into account, with the thinking that prestigious schools attract strong peers, which means that you can learn more and work with amazing people.</p>
<!--kg-card-end: markdown--><figure class="kg-card kg-image-card"><img src="https://blog.nelsonliu.me/content/images/2019/10/number_of_applications.png" class="kg-image" alt="Student Perspectives on Applying to NLP PhD Programs" loading="lazy"></figure><!--kg-card-begin: markdown--><p>There&#x2019;s also a case to be made for applying to a mixture of (1) programs that you&#x2019;re relatively confident you can be admitted to and (2) &#x201C;top choice&#x201D; programs that might have a bit more randomness in the admissions process (of course, all the schools you apply to should be places you&#x2019;d be happy going to). However, it&#x2019;s easy to be a bit too conservative when choosing where to apply&#x2014;remember that you only really need 1 offer. The majority of respondents applied to between 8 and 13 schools, though almost everyone was happy with the number of applications they submitted (Kevin, who applied to 4, thought it would have been helpful to apply to more).</p>
<p>NLP applicants in particular are lucky&#x2014;there are amazing faculty scattered around the world in a variety of different environments. Start with a large list before filtering down, and focus on finding the right fit for you personally.</p>
<h2 id="talkingtofacultybeforehand">Talking to Faculty Beforehand?</h2>
<blockquote>
<p>I did not email faculty beforehand - I don&#x2019;t think this helps (and in the case of a poorly crafted email, could actually hurt!).<br>
&#x2013; Sidd Karamcheti</p>
</blockquote>
<p>The majority of students did not email faculty before applying. Some faculty ask students to reach out&#x2014;this will usually be explicitly mentioned on their webpage. In the absence of such a notice, a reasonable policy is to not send an email.</p>
<blockquote>
<p>But that said, if you are in the vicinity of a school or doing an academic visit -- feel free to reach out to the faculty there and ask if they have a half-hour slot to meet!<br>
&#x2013; Roma Patel</p>
</blockquote>
<blockquote>
<p>I emailed one prospective advisor and asked to meet at a conference. In general, I think this is a good strategy, especially if you have research-related things to talk about with them. (Which hopefully you will, if they&#x2019;re a good advisor fit!)<br>
&#x2013; Nicholas Tomlin</p>
</blockquote>
<p>Several respondents were fortunate to meet potential future advisors at workshops or conferences / if they happened to be in the area, and found them to be quite receptive to short research meetings. It&#x2019;s good to go into these meetings with a sense of (1) what you&#x2019;d like to get out of it, and how to use this meeting effectively, (2) an awareness of their recent work, (3) a mental list of questions that you think have informative or interesting answers.</p>
<blockquote>
<p>...one of my undergrad advisors emailed a couple prospective grad advisors on my behalf, and asked them to look out for my application. I think this was particularly helpful and is maybe something worth mentioning to your undergraduate advisor.<br>
&#x2013; Nicholas Tomlin</p>
</blockquote>
<p>It is appropriate to <em>selectively</em> ignore advice about cold-emailing&#x2014;<a href="https://yonatanbisk.com/">Prof. Yonatan Bisk</a> has <a href="https://yonatanbisk.com/emailing_professors.html">a great guide that walks through the why, when, and how</a>.</p>
<p><a href="#table-of-contents">Back to the top.</a></p>
<hr>
<h1 id="statementofpurposeanamestatementofpurposehrefstatementofpurposeiclassfafalinkia">Statement of Purpose <a name="statement-of-purpose" href="#statement-of-purpose"><i class="fa fa-link"></i></a></h1>
<p>The statement of purpose is an opportunity for you to convey what you&#x2019;ve worked on and what you&#x2019;re interested in. Above all, make sure the statement is genuine and uniquely you. The &#x201C;accept/reject&#x201D; dichotomy of applications might make this process seem like a game&#x2014;leading many to believe that it&#x2019;s better to win the game (that is, be accepted) than to lose. While it&#x2019;s tempting to shape each application to say what you think faculty might want to hear, being yourself will lead to the best outcome in the end. Remember that programs and students are both looking for the right fit&#x2014;the statement is a fantastic opportunity for both sides to assess this.</p>
<blockquote>
<p>If your statement is genuine and makes clear why you want a PhD, it will resonate with the people you want it to resonate with.<br>
&#x2013; Sabrina Mielke</p>
</blockquote>
<h2 id="timelinewhentostartandfinishwriting">Timeline: When to Start and Finish Writing</h2>
<blockquote>
<p>With respect to starting writing, it is sometimes good to leave it late enough to wrap up any ongoing research projects at the end of the summer so you can write concrete things about them. For finishing writing, it&#x2019;s good to have a near-ready draft at least a month before.<br>
&#x2013; Roma Patel</p>
</blockquote>
<!--kg-card-end: markdown--><figure class="kg-card kg-image-card"><img src="https://blog.nelsonliu.me/content/images/2019/10/statement_start_date.png" class="kg-image" alt="Student Perspectives on Applying to NLP PhD Programs" loading="lazy"></figure><!--kg-card-begin: markdown--><p>Try to set aside a fixed period of time to work on your statement. While starting earlier rather than later is usually better, try to start writing a draft once you think your current projects and interests are concrete enough to write something substantive. Strive to have a preliminary draft that you&#x2019;re happy with at least a month before the deadline. You can then send this to your advisors for feedback; continue editing and iterating until the deadline and/or you&#x2019;re happy with how things look.</p>
<h2 id="structuringastatementofpurpose">Structuring a Statement of Purpose</h2>
<blockquote>
<p>The goal of the statement is to talk about your past (research) experience, and how that has prepared you for a career in research (why you&#x2019;re qualified for grad school).<br>
&#x2013; Sidd Karamcheti</p>
</blockquote>
<p>Your statement of purpose should uniquely describe your research experience and elaborate on the process you went through as you undertook your first few research projects. Give enough detail about your past work to allow them to assess the value of the work and also to concretely show that you knew what you were doing at every step of the process. Then fold this into your research as a whole. Try to leverage insights from both the actual work as well the experience of doing research, to formulate how you would undertake future projects during your graduate school career.</p>
<blockquote>
<p>Many professors do tell you what they&#x2019;re looking for in a SoP (JHU CLSP for example has hints at <a href="https://www.clsp.jhu.edu/apply-for-phd/phd-admissions-faq/">https://www.clsp.jhu.edu/apply-for-phd/phd-admissions-faq/</a>), so do use that resource.<br>
&#x2013; Sabrina Mielke</p>
</blockquote>
<h2 id="tailoringeachstatementforspecificuniversities">Tailoring Each Statement for Specific Universities</h2>
<blockquote>
<p>I only tweaked the final paragraph. In this paragraph, I specifically mentioned 2--4 faculty that I wanted to work with and provided a one sentence rationale.<br>
&#x2013; Eric Wallace</p>
</blockquote>
<p>Our survey respondents were quite divided on this question. A few respondents significantly tweaked their statements for each university to reflect the subset of their interests relevant to the prospective advisor&#x2019;s research. However most respondents kept 80-90% of their statement identical and only modified the last 1-2 paragraphs with university specific information - such as the names of the professor they were interested in working with. Most agreed that it is good to have at least some university-specific information to form a connection between your own research goals and a prospective advisor&#x2019;s research directions.</p>
<blockquote>
<p>It is good to have concrete reasons laid out in your statement as to why you want to go to this school and work with these faculty on interesting problems. So definitely tweak the section of your statement that stresses on this.<br>
&#x2013; Roma Patel</p>
</blockquote>
<h2 id="gettingfeedbackonyourstatement">Getting Feedback on Your Statement</h2>
<blockquote>
<p>Your recommenders will get a better sense of your research interests so it can help them write your recommendation and they have also been through similar processes.<br>
&#x2013; Kevin Lin</p>
</blockquote>
<p>It is good to have a near-complete draft of your statement ready in time to send to your recommenders before they begin to write your letter of recommendation. There are multiple benefits to this. Reading your statement will help them better understand your research interests, which will not only allow them to concretely write things about you in their letter, but might also bring up useful pieces of advice from them based on what they know of the people working in that research area. They will also usually give you feedback on the overall statement&#x2014;they have possibly read countless statements over the course of their career and will be able to fairly judge and evaluate this in context. Your research advisors and recommenders are likely both extremely knowledgeable and also have your best interests at heart, so remember to ask for feedback and advice on your application!</p>
<h2 id="usingthisasalearningopportunity">Using this as a Learning Opportunity</h2>
<blockquote>
<p>In my statement, I mostly talked about my past experiences and how they feed into my current research interests. I tried to paint a picture that enables the reader to better understand how I reached / why I do the research I do.<br>
&#x2013; Nelson Liu</p>
</blockquote>
<p>Write out your journey as a researcher from the beginning to the present. This will convey important information about you and your research, which can be illuminating for both your reader and for yourself. Chances are that you will write dozens of similar statements in the future, whether they are research statements for fellowships, project proposals, or grant applications. Use this as a learning experience! Writing your statement of purpose is not only good practice for the future, but also a rare invitation to reflect upon your interests and motivations.</p>
<p><a href="#table-of-contents">Back to the top.</a></p>
<hr>
<h1 id="lettersofrecommendationanamelettersofrecommendationhreflettersofrecommendationiclassfafalinkia">Letters of Recommendation <a name="letters-of-recommendation" href="#letters-of-recommendation"><i class="fa fa-link"></i></a></h1>
<p>Letters of recommendation are <a href="https://www.cs.cmu.edu/~harchol/gradschooltalk.pdf">often cited</a> as the most important part of a PhD application. In our survey, every respondent marked letters as either the most or second-most important component. Given that the admissions committee is optimizing to admit candidates with a high likelihood of reliably producing excellent research, a letter from a fellow academic that effectively claims you&#x2019;ve been able to do so is a strong signal that you&#x2019;re a good candidate.</p>
<h2 id="whattolookforwhenchoosingletterwriters">What to look for when choosing letter writers</h2>
<blockquote>
<p>Your letter writers should be people who know you well enough to speak about your skills and your strengths as a PhD candidate ... people you have worked with who are doing relevant research in the field and people you have genuinely been advisors to you&#x2026;<br>
&#x2013; Roma Patel</p>
</blockquote>
<p>It can be helpful to view letter writers as your <strong>primary advocates</strong> in the admissions process. They want their excellent undergraduate students or research assistants to succeed, and they&#x2019;re singing your praises in order to argue for your spot in graduate school. From this view, it may be clear that they should know you, your strengths, and your goals. Of course, some of your letter writers will know you better than others, but each should be able to at least advocate for your excellence in how you worked or interacted with them.</p>
<p>There&#x2019;s often a tradeoff between (1) how well you know the letter writer, (2) how cool the work you did with them was, and (3) how well-known they are. As a first approximation, attempt to have all 3 letter writers know you through some kind of research collaboration. Simply doing well in their class, or TAing for them does not necessarily make for a strong letter. On the other hand, an industry researcher who can vouch for your research ability may be able to make a stronger statement. This brings us to (3) how well known the letter-writer is. Perhaps unfortunately, letters from well-known members of the field are (very) highly regarded. This may be due to fame bias&#x2014;the professors on the application committee can rest assured that they know so-and-so from X university consistently recommends only excellent students.  As suggested at the beginning of this paragraph, this will play some role in the tradeoff, but keep in mind that a famous professor who doesn&#x2019;t really know you won&#x2019;t write a strong letter.</p>
<p>Each of the components mentioned above&#x2014;personal knowledge of you and your work, successful research and fame of the writer were mentioned by our respondents.</p>
<blockquote>
<p>I chose professors with whom I had completed somewhat successful research, and who were likely to be known by my prospective advisors. For better or worse (probably worse), connections between letter writers and prospective advisors seem to matter a lot.<br>
&#x2013; Nicholas Tomlin</p>
</blockquote>
<h2 id="whentostartlookingforrecommenders">When to start looking for recommenders</h2>
<p>People get started in research at different times, but by the time of application, you need three people who can advocate for your spot in graduate school (though again, not all need to be equally strong or know you equally well). When should you start building these relationships? The easy answer is &#x201C;as early as possible&#x201D;. Research takes a long time, as does getting settled in a field and starting to make real progress. This creates a definite bias towards those who start research earlier and collaborate widely (3 professors means a lot of connections to make). However, everyone&#x2019;s research story looks different, and <strong>no student should think it &#x201C;too late&#x201D; to go for a PhD</strong> (though a master&#x2019;s and/or further years of research experience may be necessary.)</p>
<p>To back this up, note the wide range of times that our respondents started working with the people who would end up being their LoR writers.</p>
<!--kg-card-end: markdown--><figure class="kg-card kg-image-card"><img src="https://blog.nelsonliu.me/content/images/2019/10/lor_writers_start_date.png" class="kg-image" alt="Student Perspectives on Applying to NLP PhD Programs" loading="lazy"></figure><!--kg-card-begin: markdown--><p>Note that this histogram includes one data point for each letter writer for each respondent. (Not everyone mentioned all three writers, and one mentioned four.) I counted &#x201C;summer before 3rd year&#x201D; as &#x201C;2nd year.&#x201D; That&#x2019;s a lot of letter writers from the third and fourth (!) years. Many respondents who met their letter writers after their third year did indicate that it would have been better to start earlier, but the data somewhat makes sense&#x2014;as you progress through your studies, you gain more research experience.</p>
<h2 id="askingforspecificsinyourletterandgettingthemsubmitted">Asking for specifics in your letter, and getting them submitted</h2>
<p>Recall that your letter writers are your advocates&#x2014;you should feel empowered to bring up all the awesome things that you did with them, and ask (but not demand) that they mention specific things. These requests may be to tailor their letters to your statement of purpose. Think that your efforts in conducting replicable science in a world of AI hype are awesome? Your letter writer may agree, but likely wouldn&#x2019;t think to mention it if you don&#x2019;t remind them.</p>
<blockquote>
<p>I made sure to send a reminder email 2 weeks, then 1 week, then a few days before applications were due.<br>
&#x2013; Nelson Liu</p>
</blockquote>
<p>Likewise, remember that they&#x2019;re human and busy, and very well may forget your letter if you don&#x2019;t send them a few reminders. PhD applications tend to have lenient letter of recommendation deadlines but it&#x2019;s better to keep on top of them with tastefully-spaced reminder emails&#x2014;better to not test the waters in this context.</p>
<p><a href="#table-of-contents">Back to the top.</a></p>
<hr>
<h1 id="publicationsanamepublicationshrefpublicationsiclassfafalinkia">Publications <a name="publications" href="#publications"><i class="fa fa-link"></i></a></h1>
<blockquote>
<p>I think that having a published conference paper greatly increases your chances, but I think that papers are merely a signal for something more important: can you complete the full research process, from idea inception to experiment execution to writing things up?<br>
&#x2013; Nelson Liu</p>
</blockquote>
<p>Most respondents felt that publications are an important part of a strong application, but are not necessary if you have stellar recommendation letters talking about your research aptitude. Admission into PhD programs in computer science (especially at top schools) is quite competitive, and many candidates have publications, especially candidates applying after year-long research positions such as AI residency programs.</p>
<blockquote>
<p>Publications are just tangible evidence - if you can show other evidence that you are able to do research, that you learned something, that you have skills/conclusions that you&#x2019;ve taken away from the experience, then you should be fine.<br>
&#x2013; Sidd Karamcheti</p>
</blockquote>
<p>Publications are a good way to show concrete research output. This acts like &#x201C;hard evidence&#x201D; of research aptitude, which is the primary criterion used to judge PhD applicants. Alternative ways to show concrete research output could be excellent research code releases or insightful blog posts.</p>
<p><a href="#table-of-contents">Back to the top.</a></p>
<hr>
<h1 id="transcriptsgradesanametranscriptsgradeshreftranscriptsgradesiclassfafalinkia">Transcripts / Grades <a name="transcripts--grades" href="#transcripts--grades"><i class="fa fa-link"></i></a></h1>
<p>Almost all survey respondents thought that grades and GPA scores play only a minor role in NLP PhD admissions. It is wise to not stress too much about improving your GPA, especially if compromises the time spent doing research. Things might be different in more theoretical fields though, where coursework might be closer to research.</p>
<blockquote>
<p>Take an intro to NLP course! Take machine learning or a specific linguistics course or anything else that clearly shows that you have studied the topics you are excited about in depth.<br>
&#x2013; Roma Patel</p>
</blockquote>
<blockquote>
<p>Interesting classes off the beaten path may let you stand out from the crowd.<br>
&#x2013; Sabrina Mielke</p>
</blockquote>
<p>The choice of coursework typically acts like a skillset evaluation during PhD admissions, checking whether candidates are familiar with the fundamental techniques required to conduct their research. Coursework can also help present a coherent academic history when combined with the statement of purpose. Some courses might help an applicant stand out from the crowd, especially if they&#x2019;re uniquely relevant or off the beaten path.</p>
<blockquote>
<p>Sometimes, the exact preparation matters less than evidence that you&#x2019;re capable of learning important background material. E.g., despite me not having strong probability/stats background, a few professors said they were impressed by my (completely irrelevant) pure math background.<br>
&#x2013; Nicholas Tomlin</p>
</blockquote>
<p>While coursework does not play a major role in admission decisions, many respondents mentioned that courses are a great way to learn the fundamentals and get interested in a particular field, often acting like a precursor to research.</p>
<p><a href="#table-of-contents">Back to the top.</a></p>
<hr>
<h1 id="standardizedexamsgretoeflanamestandardizedexamsgretoeflhrefstandardizedexamsgretoefliclassfafalinkia">Standardized Exams: GRE / TOEFL <a name="standardized-exams-gre--toefl" href="#standardized-exams-gre--toefl"><i class="fa fa-link"></i></a></h1>
<blockquote>
<p>I get the sense that the GRE doesn&#x2019;t really matter unless you do abysmally.<br>
&#x2013; Nelson Liu</p>
</blockquote>
<p>Nearly everyone agreed that scores from required standardized tests are not deal-breaking as long as you meet a minimum threshold. Having a suspiciously low score could raise questions, but barring failing the exam, this should not significantly impact your entire application. That said, this is a required checkpoint on your application, so keep aside time to get this done correctly.</p>
<blockquote>
<p>There is no glory or shame in taking too much or too little time, so it is better to not compare to others and keep aside the right (and possibly minimal) amount of time you think you need to prepare.<br>
&#x2013; Roma Patel</p>
</blockquote>
<p>Try to give yourself at least 1-2 weeks of study time before the actual test. Don&#x2019;t consider the amount of time you see others spending on this &#x2014; assess yourself and allocate larger amounts of time to topics that you are uncertain about and think could use the extra effort. Remember to review all the topics you need to, take a few practice tests, and then just take the exam and don&#x2019;t stress about the score.</p>
<p>It is usually not worth the extra time, effort, cost (or effect) to redo the exam. So prepare well once, take the exam, and don&#x2019;t stress about the score once you are done with it. For what it&#x2019;s worth, future years will likely see this disregard and ambivalence towards scores on tests heightened &#x2014; lots of schools have already removed the GRE requirement, while others have definite plans of doing so in the coming years.</p>
<p>In general, international students must submit their TOEFL (or IELTS) scores to demonstrate competency in the English language &#x2014; however for some schools, international students who have received degrees in US schools or received their instruction in English do not need to submit TOEFL scores. Unlike in GRE, applicants MUST score higher than the minimum requirements if universities sets minimum scores. The minimum requirements vary from program to program. For example, the Cornell CS PhD program sets the minimum scores for each section (Listening 15, Writing 20, Reading 20, Speaking 22), while the MIT EECS PhD set the total minimum scores to 100. Make sure that you meet TOEFL scores before the application deadline. Unfortunately, the applicants whose TOEFL scores lower than the minimum are likely to be &#x201C;desk-rejected&#x201D;.</p>
<p><a href="#table-of-contents">Back to the top.</a></p>
<hr>
<h1 id="interviewspostapplicationcallsanameinterviewspostapplicationcallshrefinterviewspostapplicationcallsiclassfafalinkia">Interviews / Post-application Calls <a name="interviews--post-application-calls" href="#interviews--post-application-calls"><i class="fa fa-link"></i></a></h1>
<blockquote>
<p>Interviews in USA are less formal - more general discussions about research interests. Interviews for Europe in my experience were more in depth, as they expect you to already have knowledge of your field (since you can only apply after a Masters), have a research plan and expect you to have already surveyed literature in your chosen field of interest.<br>
&#x2013; Aishwarya Kamath</p>
</blockquote>
<p>The interviews and visit days will differ significantly over the range of schools you&#x2019;re considering&#x2014;both in their intended purpose and in the amount of information you can glean about the school and faculty from this one interaction. Some schools do pre-acceptance visit days, with offers conditioned on the interviews and ensuing discussions. Others do virtual interviews over the phone or video calls. And of course, some schools choose not to conduct interviews.</p>
<p>While each interview experience is largely dependent on the candidate in question, most of our survey respondents agreed that these conversations follow the same general pattern.</p>
<blockquote>
<p>The general format was like:</p>
<ol>
<li>&#x201C;Tell me about a research project you worked on (pick one that is most exciting and introduce)&#x201D;. The professor would ask some questions, like &#x201C;why did you consider this model / run this experiment?&#x201D;, &#x201C;what is the conclusion?&#x201D;, &#x201C;what did you learn through this project?&#x201D;</li>
<li>&#x201C;What is your research interest?&#x201D;, &#x201C;What are you interested in doing for your PhD (and your career)?&#x201D; -- it&#x2019;s good to think in both short term and long term</li>
<li>&#x201C;Do you have any questions?&#x201D; -- you can ask any questions about the lab, like the culture, research goals, how advising/meeting works.<br>
&#x2013; Michi Yasunaga</li>
</ol>
</blockquote>
<p>This is mostly a means of trying to get a sense of what you are like as a person and what your research interests are, to assess both compatibility and mutual interests. Your interviewers will generally ask you to talk about the research you have done &#x2014; and will interrupt with questions about things that they are interested to hear more about. Overall, this is less of an assessment of your knowledge, rather than them getting insights into how you solve problems and talk about research.</p>
<blockquote>
<p>I didn&#x2019;t enjoy the whiteboard interview.<br>
&#x2013; Nicholas Tomlin</p>
</blockquote>
<p>This sometimes happens. If professors want to assess a specific component of your application, or want to know the extent of your knowledge about a certain topic, they will ask you technical questions that can range from explaining or solving an algorithm, writing out equations or explaining computational and implementation-specific aspects of things you have done. Most of our survey applicants however, did not have to go through this and their interviews largely consisted of general research conversations.</p>
<blockquote>
<p>You should definitely know your own work inside-out, but don&#x2019;t stress about having to know every intricate detail about every subfield in NLP.<br>
&#x2013; Roma Patel</p>
</blockquote>
<p>While it is not important (or even possible) to know everything little thing about every research area in NLP, you should be aware of work being done in areas related to you. Most importantly, if you have written about something in your statement, you should be able to confidently speak about it and answer any questions that they throw at you. Take time to look into every detail and ensure that you know the fundamentals of your work before your interview.</p>
<blockquote>
<p>Remember that this is a two way street&#x2014;while they&#x2019;re assessing whether you&#x2019;d be a good fit for their program, you should be probing whether this place / professor is a good match for you.<br>
&#x2013; Nelson Liu</p>
</blockquote>
<p>There is usually a part of the interview where the interviewer steps back and asks you to ask questions &#x2014; use this time to probe at any uncertainties or lingering questions that you have. If you have questions about their previous work, thoughts about future possibilities, or even just general questions about the program or the department, use this time to clear any doubts and get all the answers you will need to make a decision.</p>
<blockquote>
<p>if you don&#x2019;t know something, it is okay to say that you don&#x2019;t --- ask questions that help you understand it more and treat it as a learning experience.<br>
&#x2013; Roma Patel</p>
</blockquote>
<blockquote>
<p>The only thing I will tell you not to do in an interview: pretend. Professors are good at spotting that kind of thing and they will strongly judge you for it. Just be honest and genuine. You are starting your PhD. You don&#x2019;t need to know things -- just be willing to grow.<br>
&#x2013; Sabrina Mielke</p>
</blockquote>
<p>Also, don&#x2019;t worry if you do not know everything the interviewers ask. Just try to be as honest and genuine as you can, and show that you are willing to learn and grow, instead of pretending to know the topics.</p>
<blockquote>
<p>I think the interviews as an initial conversation really affected where I seriously considered&#x2014;the places with interviews that I thought were more fair / reasonable gained legitimacy. In the best case, it was basically a research conversation with a senior researcher, and a great opportunity to get feedback / hear what they think about the field. Overall, I thought they were quite valuable, and I wish that I had treated them less as assessments and more as opportunities.<br>
&#x2013; Nelson Liu</p>
</blockquote>
<p>Make the most of your interviews! All applicants agreed that overall, the interviews were friendly and engaging experiences. Think of this as an opportunity to speak about and answer questions about your work and to have a mutually engaging research conversation.</p>
<blockquote>
<p>One useful piece of advice from one of my undergrad advisors was to, &#x201C;Talk about your research ideas! Remember that what most faculty really want is to be able to discuss the research that is important to them &#x2014; and if you can do this and make exciting progress through these discussions, you will both mutually have a productive and happy career together.&#x201D;<br>
&#x2013; Roma Patel</p>
</blockquote>
<p><a href="#table-of-contents">Back to the top.</a></p>
<hr>
<h1 id="decidingwheretogoanamedecidingwheretogohrefdecidingwheretogoiclassfafalinkia">Deciding where to go <a name="deciding-where-to-go" href="#deciding-where-to-go"><i class="fa fa-link"></i></a></h1>
<p>If you&#x2019;re fortunate to be considering multiple options, congratulations! It is a hard problem, but a good one to have&#x2014;be aware of your privilege. The choice between graduate programs is an intensely personal one, and there are a variety of academic and non-academic factors to consider, all of which will influence your health, happiness, and productivity.</p>
<blockquote>
<p>Something that people do not always remember when making a decision is that your advisor is possibly someone you will be talking to for upto 3 hours every week for nearly 6 years of your life. It is good to rethink whether or not you will be happy doing this with the faculty in question, if the two of you see eye-to-eye, can comfortably talk about both research-things and also life-things when they come up, and that they will encourage and help guide you in everything you need to do the research that is important to you during your PhD.<br>
&#x2013; Roma Patel</p>
</blockquote>
<p>In general, most respondents agreed that the most important factor is your primary advisor&#x2014;who will you be working with during your PhD? Do you have mutual research interests? Are your communication and working styles compatible? Would you be comfortable talking to them about your struggles, both academic and non-academic? Do you have much to learn from them and their group? Do you feel supported by them? While it is hard to assess these deep questions before spending time to work with them, conversations and interactions during visit days will help you get a sense of whether things feel right. Trust your instinct&#x2014;if things feel odd or unnatural, even during these initial conversations, you have plenty of reason to reconsider and be hesitant.</p>
<blockquote>
<p>As an undergrad at a school with a large NLP community, I really benefited from having senior researchers around (e.g., grad students and postdocs)---I have so much to learn from them! I felt like I wanted to keep having such an environment in graduate school, which actually ended up being one of the defining factors in my final choice.<br>
&#x2013; Nelson Liu</p>
</blockquote>
<p>Many students also took note of the NLP community at every school they were considering. For instance, some prefer larger groups with many senior students and postdocs, while others prefer smaller, more-intimate groups. There are benefits and drawbacks to both sorts of research environments, and it ultimately boils down to personal preference and taste. It&#x2019;s important that you feel like you have enough people around to talk about research and life&#x2014;while your advisor is an important figure in the PhD, you will spend the majority of your time talking to and working alongside fellow students. Make sure that these are people that you&#x2019;d love to be around for the next stage of your research career.</p>
<blockquote>
<p>Sure, you&#x2019;re picking a place to do research for the next 5+ years of your life, but you also need to be happy / have a life outside of research...I went climbing during a lot of my visits, mostly to assess convenience.<br>
&#x2013; Nelson Liu</p>
</blockquote>
<p>Another important factor to consider is the location. Several expressed weather / culture preference (mostly on the east-coast-vs-west-coast divide). Many also wanted to be in a place that was affordable for students and conveniently located to their favorite hobbies or recreational activities. While research fit is certainly important, you won&#x2019;t be productive if you&#x2019;re miserable&#x2014;put your happiness and your health first, and make sure that you&#x2019;ll be happy as both a student on-campus and as a resident of the area.</p>
<blockquote>
<p>Prestigious schools attract strong peers, which means you can learn more and collaborate with amazing people.<br>
&#x2013; Eric Wallace</p>
</blockquote>
<p>Several also considered the relative &#x201C;ranking&#x201D; of a university or program (though this is almost impossible to objectively evaluate without implicitly considering the other factors). While rankings can tell part of the story, they&#x2019;re not substitute for your own feelings and intuitions about where you belong.</p>
<blockquote>
<p>At some schools, it was very clear who my advisors would be, while at others, it wouldn&#x2019;t be decided until I&#x2019;d enrolled. I preferred the former scenario since it involved less uncertainty.<br>
&#x2013; Lucy Li</p>
</blockquote>
<p>It&#x2019;s also useful to consider the program&#x2019;s requirements and logistics around advising. Are you guaranteed to be able to work with the advisor(s) you are interested in? Does the department have extensive qualification exams or requirements that might be hindrances to your productivity? Will you have to worry about funding?</p>
<blockquote>
<p>Personal feelings actually do matter. If you feel (even slightly) uncomfortable, these negative feelings will grow during the five years.<br>
&#x2013; Akari Asai</p>
</blockquote>
<blockquote>
<p>Once you have done an extensive comparison on all parameters (professional and personal), you might be stuck between 2-3 very good options. Try reweighting the parameters and see if the balance shifts towards one end. If you are still confused, don&#x2019;t worry :) If it&#x2019;s so confusing, both places are surely very good. You will need to work very hard wherever you go, and you won&#x2019;t lose much choosing one over the other. Go with your heart.<br>
&#x2013; Kalpesh Krishna</p>
</blockquote>
<p>When it comes to the final decision, everyone agrees to go with your heart and feelings of what seems right to you. We&#x2019;re all logical and analytical people (perhaps to a fault), but if you can&#x2019;t make up your mind about where to go / are stuck between several options, pick the one that you feel the best about inside. One way to discern this: Suppose you&#x2019;re picking between two places (this strategy generalizes to N). Take a coin, and assign one place to heads and another to tails. Tell yourself that the result of the coin flip will be where you end up going. Flip the coin, and observe the result. Are you relieved? Would you have preferred the other side? The answer to these questions might help you better understand how you really feel about the decision.</p>
<blockquote>
<p>Whatever you do end up deciding, though, don&#x2019;t regret it&#x2014;the decision is done now, and you just have to put in the work to ensure that it is a good one.<br>
&#x2013; Nelson Liu</p>
</blockquote>
<h2 id="makingthemostofvisitdays">Making the most of visit days</h2>
<blockquote>
<p>I didn&#x2019;t end up going to most visit days -- which is not something that you should do. Go to every visit day! Talk to the other students visiting, the other students currently pursuing PhDs there and to the faculty there. Keep a list of standard questions about schools (requirements, professors, exams, time taken) and make a note of these for every school so that you have an easy way to compare at decision-making time.<br>
&#x2013; Roma Patel</p>
</blockquote>
<p>Many of our survey respondents recommend making the most of the visit days. Treasure this priceless opportunity to talk to professors (both in and outside of your field), meet PhD students, and get to know the other students in your cohort. As you continue your academic career, you&#x2019;ll be seeing all of these people around in the future&#x2014;get to know them now!</p>
<blockquote>
<p>Talk to students most of all -- disturb them when they&#x2019;re working to see what it&#x2019;s like in the lab!<br>
&#x2013; Sabrina Mielke</p>
</blockquote>
<p>Before each visit, it&#x2019;s useful to think a bit about what you&#x2019;d like to get out of it. This might result in a list of questions you&#x2019;d like to answer, or people that you&#x2019;d like to talk to. Don&#x2019;t be afraid to contact PhD students in the department and ask to meet; the majority are happy to do so, and would love to give you advice, hear about what you&#x2019;re working on, and talk about their research. Talking to students is of the utmost importance; they will tell you what it&#x2019;s really like in the department, and it&#x2019;s useful for getting a sense of the overall department culture and graduate student community.</p>
<blockquote>
<p>My advisor, in her infinite wisdom, gave me a useful piece of insight that had not struck me before. &quot;What most people don&apos;t realise, is that the people that you are meeting and talking to over these visits will likely be in your life, for the rest of your life. Go to as many visits and talk to as many prospective students as you can &#x2014; some of your closest friends and advisors will come out of these interactions.&quot;<br>
&#x2013; Roma Patel</p>
</blockquote>
<p><a href="#table-of-contents">Back to the top.</a></p>
<hr>
<h1 id="misctopicsanamemisctopicshrefmisctopicsiclassfafalinkia">Misc. Topics <a name="misc-topics" href="#misc-topics"><i class="fa fa-link"></i></a></h1>
<h2 id="residencyprogramsasprecursorstoyourphd">Residency Programs as Precursors to your PhD</h2>
<blockquote>
<p>I see a couple benefits of working in AI residency which I did at AI2. 1) if you aren&#x2019;t sure if you want to do a PhD, this is a pretty good way to find out, and after the residency you will be in a reasonable position to pursue both industrial and PhD positions. 2) You will be exposed to a new set of people, and it is helpful to learn from different ways of doing research 3) I personally changed my research direction towards more NLP and this was a great way to explore different research topics and build up the skills I needed to pursue those topics.<br>
&#x2013; Kevin Lin</p>
</blockquote>
<blockquote>
<p>Be really really clear why you&#x2019;re doing the residency - the reason to do the residency/work is to do something you could not otherwise do at grad school/if you&#x2019;re not sure about grad school.<br>
&#x2013; Sidd Karamcheti</p>
</blockquote>
<p>It&#x2019;s really important to consider why you want to do a residency program. As our survey respondents mentioned, there are a few different paths that lead to residencies&#x2014;foremost among them is if you&#x2019;re not too sure about wanting to do a PhD, and you want some more research experience (working with a couple of different mentors with possibly different areas/interests than what you were exposed to as an undergraduate) before making a final decision.</p>
<p>Another reason a residency program is a good idea is if you&#x2019;re sure about doing a PhD, but had limited exposure to different areas as an undergraduate. Especially if you&#x2019;re considering PhD programs where you&#x2019;re paired with an advisor/placed in a specific area outright, having a year to explore a bunch of different areas and work with different mentors with different styles will let you make a more informed decision. It&#x2019;s totally possible that the residency program will introduce you to areas you would never have otherwise considered!</p>
<p>That being said, it&#x2019;s worth noting that not all residency opportunities are created equal&#x2014;several different companies are just in their first or second year of offering their residency programs, meaning that they&#x2019;re subject to growing pains&#x2014;without structured onboarding/tutorials you might spend a lot of time trying to figure out how to use company infrastructure, or you might spend a lot of time trying to figure out what different folks at the company are working on, and how research works in industry.</p>
<p>More importantly, you need to make sure your residency mentors are committed to the same goals that you are&#x2014;a mismatch in expectations between you and your residency mentors is going to significantly sour your experience! If you want to explore a bunch of different sub-areas of your chosen research area, make sure your mentor is on board to try a few different projects over the course of the year! If you want to instead work on more long-term projects/existing initiatives at the company, make sure that your host is willing to connect you with these existing teams, and that there&#x2019;s some structure in place that will let you (1) learn, and (2) contribute.</p>
<p>Finally, don&#x2019;t feel like you need to do a residency to get the industry experience, or to explore different research areas. There is definitely a large amount of time you can spend exploring different areas in grad school, and you&#x2019;ll have multiple summers to do internships where you&#x2019;ll possibly get to work on projects very different from your core research agenda.</p>
<blockquote>
<p>FWIW, you will likely intern at a lot of the places during the course of your PhD and will have a similar experience, so if the only reason you are considering a residency is because you think that is an experience you will never get at a later time --- this is likely not true.<br>
&#x2013; Roma Patel</p>
</blockquote>
<blockquote>
<p>When submitting my application, I was pretty sure that I would defer for a year if I got an offer---there&#x2019;s no rush, and the extra year might give me some interesting perspective.<br>
&#x2013; Nelson Liu</p>
</blockquote>
<p><a href="#table-of-contents">Back to the top.</a></p>
<hr>
<h1 id="inconclusionanameinconclusionhrefinconclusioniclassfafalinkia">In Conclusion <a name="in-conclusion" href="#in-conclusion"><i class="fa fa-link"></i></a></h1>
<p>If you&#x2019;ve read this far, we hope that this discussion was useful. The admissions process is inherently stochastic, and there&#x2019;s much that you can&#x2019;t control&#x2014;relax, have confidence in yourself, and goodluck!</p>
<blockquote>
<p>Another good advice I received from my friend was &#x201C;Don&#x2019;t reject (by?) yourself&#x201D;. I remember how uneasy and stressful I felt at the time of application, as I did not have strong publication records, and came from non top undergraduate schools in the US. Sometimes people value your unique back-ground, experience in other fields or find really positive signals in the letters of recommendation. Don&#x2019;t hesitate to apply for good schools, because &#x201C;I think I&#x2019;m not good enough&#x201D;.<br>
&#x2013; Akari Asai</p>
</blockquote>
<p><a href="#table-of-contents">Back to the top.</a></p>
<!--kg-card-end: markdown-->]]></content:encoded></item><item><title><![CDATA[Software Archaeology: Re-generating the CoNLL 2000 Chunking Data]]></title><description><![CDATA[<!--kg-card-begin: markdown--><p>I&apos;ve been using the data from the <a href="https://www.clips.uantwerpen.be/conll2000/chunking/">CoNLL 2000 shared task on syntactic chunking</a> for some ongoing work, but the original dataset is tiny by modern standards. The train set is sections 15-18 of the Penn Treebank, and the test set is section 20---there is no development split.</p>]]></description><link>https://blog.nelsonliu.me/2018/10/28/re-generating-conll-2000-chunking-data/</link><guid isPermaLink="false">5d5962bc2aa85523bca5b522</guid><category><![CDATA[research]]></category><category><![CDATA[tooling]]></category><category><![CDATA[tutorial]]></category><category><![CDATA[open source]]></category><category><![CDATA[software archaeology]]></category><category><![CDATA[nlp]]></category><dc:creator><![CDATA[Nelson Liu]]></dc:creator><pubDate>Sun, 28 Oct 2018 00:43:53 GMT</pubDate><content:encoded><![CDATA[<!--kg-card-begin: markdown--><p>I&apos;ve been using the data from the <a href="https://www.clips.uantwerpen.be/conll2000/chunking/">CoNLL 2000 shared task on syntactic chunking</a> for some ongoing work, but the original dataset is tiny by modern standards. The train set is sections 15-18 of the Penn Treebank, and the test set is section 20---there is no development split.</p>
<p>Since my specific application doesn&apos;t need to be comparable to past work and models on the task, I set about re-generating the data from a larger portion of the Penn Treebank. This was more involved than anticipated, maybe because the data and task are so old---I had to do a bit of software archaeology, and the steps are detailed below.</p>
<blockquote class="twitter-tweet" data-cards="hidden" data-lang="en"><p lang="en" dir="ltr">If you ever do a career in science with computers, you&apos;ll be doing software archeology more often than you might think: rewamping old code/simulations/analysis to work in new environments. <a href="https://t.co/4ZI2qOnwEe">pic.twitter.com/4ZI2qOnwEe</a></p>&#x2014; Gael Varoquaux (@GaelVaroquaux) <a href="https://twitter.com/GaelVaroquaux/status/952994313692164096?ref_src=twsrc%5Etfw">January 15, 2018</a></blockquote>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<h2 id="step1sourcethescriptusedtogeneratethedata">Step 1: Source the script used to generate the data</h2>
<p>The CoNLL 2000 shared task site helpfully notes:</p>
<blockquote>
<p><a href="http://ilk.uvt.nl/team/sabine/homepage/software.html">http://ilk.uvt.nl/team/sabine/homepage/software.html</a><br>
The Perl script that was used for generating these training and test data sets from the Penn Treebank. It has been written by Sabine Buchholz from Tilburg University.</p>
</blockquote>
<p>However, following the link and proceeding to the script download results in a dead link. Perhaps expected, since it&apos;s been almost 20 years.</p>
<p>By searching for the filename on GitHub (a great tool for finding old software and scripts), I stumbled upon <a href="https://github.com/mgormley/concrete-chunklink">this repo</a> from Matt Gormley that has a modified version of the <code>chunklink</code> Perl script. Here&apos;s a gist to the script for posterity: <a href="https://gist.github.com/nelson-liu/4a1872d7062868cbc1affb545710b836">https://gist.github.com/nelson-liu/4a1872d7062868cbc1affb545710b836</a></p>
<h2 id="step2runthescriptusedtogeneratethedata">Step 2: Run the script used to generate the data.</h2>
<p>Perl was before my time, but I managed to run the script with the <code>perl</code> on my Macbook. Here&apos;s the output of <code>perl -v</code></p>
<pre><code>$ perl -v

This is perl 5, version 18, subversion 2 (v5.18.2) built for darwin-thread-multi-2level
(with 2 registered patches, see perl -V for more detail)
</code></pre>
<p>To run the script, I downloaded the Penn Treebank and wrote a quick bash script to invoke the script on each Penn Treebank section in turn, redirecting the output for each section to a file.</p>
<p>The files generated by <code>chunklink_2-2-2000_for_conll.pl</code> are not in the CoNLL 2000 format, so I wrote a separate Python script called <code>convert_to_conll2000_format.py</code> to massage the output into proper space-sparated CoNLL chunking format. You can download that script here: <a href="https://gist.github.com/nelson-liu/4faaf5ccc67636939b299b289720ea94">https://gist.github.com/nelson-liu/4faaf5ccc67636939b299b289720ea94</a> , and it should be Python 2.x / 3.x compatible.</p>
<pre><code class="language-bash">#! /usr/bin/env bash
set -e

# Untar the raw PTB data
echo &quot;Unzipping raw PTB data&quot;
tar -xf treebank_3_LDC99T42.tgz

# Make chunking data for each PTB section
mkdir -p chunklink_generated_data
mkdir -p conll2000_data
for section_num in {00..24}
do
    echo &quot;Creating chunking data for section ${section_num}&quot;
    cat treebank_3/parsed/mrg/wsj/${section_num}/*.mrg | perl chunklink_2-2-2000_for_conll.pl -N -ns &gt; chunklink_generated_data/${section_num}.chunklink
    python convert_to_conll2000_format.py chunklink_generated_data/${section_num}.chunklink &gt; conll2000_data/${section_num}.conll
done
</code></pre>
<p>This produces a folder named <code>conll2000_data</code> with <code>00.conll</code>, <code>01.conll</code>, etc. with the ConLL 2000-formatted data for each of the Penn Treebank sections. You can use <code>cat</code> to combine sections and create whatever train, dev, and test splits you might want.</p>
<p>Happy chunking!</p>
<!--kg-card-end: markdown-->]]></content:encoded></item><item><title><![CDATA[Extracting last timestep outputs from PyTorch RNNs]]></title><description><![CDATA[Here's some code I've been using to extract the last hidden states from an RNN with variable length input.]]></description><link>https://blog.nelsonliu.me/2018/01/25/extracting-last-timestep-outputs-from-pytorch-rnns/</link><guid isPermaLink="false">5d5962bc2aa85523bca5b521</guid><category><![CDATA[research]]></category><category><![CDATA[tooling]]></category><category><![CDATA[tutorial]]></category><category><![CDATA[machine learning]]></category><category><![CDATA[nlp]]></category><category><![CDATA[pytorch]]></category><dc:creator><![CDATA[Nelson Liu]]></dc:creator><pubDate>Thu, 25 Jan 2018 05:05:51 GMT</pubDate><content:encoded><![CDATA[<!--kg-card-begin: markdown--><p>Here&apos;s some code I&apos;ve been using to extract the last hidden states from an RNN with variable length input. In the code example below:</p>
<ul>
<li><code>lengths</code> is a <em>list</em> of length <code>batch_size</code> with the sequence lengths for each element in the batch. It&apos;s a list because <code>pack_padded_sequence</code> also takes a list, so you already have it probably lying around.</li>
<li><code>batch_first</code> is a boolean indicating whether the RNN is in <code>batch_first</code> mode or not.</li>
<li><code>output</code> is the output of a PyTorch RNN as a <code>Variable</code>. If your output isn&apos;t a <code>Variable</code> for some reason, just remove the <code>Variable</code> call in the last line on <code>idx</code>.</li>
</ul>
<pre><code class="language-python">idx = (torch.LongTensor(lengths) - 1).view(-1, 1).expand(
    len(lengths), output.size(2))
time_dimension = 1 if batch_first else 0
idx = idx.unsqueeze(time_dimension)
if output.is_cuda:
    idx = idx.cuda(output.data.get_device())
# Shape: (batch_size, rnn_hidden_dim)
last_output = output.gather(
    time_dimension, Variable(idx)).squeeze(time_dimension)
</code></pre>
<p>Here&apos;s a full code example with a RNN and variable-length input, adapted from <a href="https://discuss.pytorch.org/t/simple-working-example-how-to-use-packing-for-variable-length-sequence-inputs-for-rnn/2120/14">an example on the PyTorch forums</a>:</p>
<pre><code class="language-python">import torch
from torch.autograd import Variable
import torch.nn as nn

batch_size = 4
max_length = 3
hidden_size = 2
n_layers = 1
input_dim = 1
batch_first = True

# Data
vec_1 = torch.FloatTensor([[1, 2, 3]])
vec_2 = torch.FloatTensor([[1, 2, 0]])
vec_3 = torch.FloatTensor([[1, 0, 0]])
vec_4 = torch.FloatTensor([[2, 0, 0]])

# Put the data into a tensor.
batch_in = torch.zeros((batch_size, max_length, input_dim))
batch_in[0] = vec_1
batch_in[1] = vec_2
batch_in[2] = vec_3
batch_in[3] = vec_4

# Wrap RNN input in a Variable. Shape: (batch_size, max_length, input_dim)
batch_in = Variable(batch_in)
# The lengths of each example in the batch. Padding is 0.
lengths = [3, 2, 1, 1]

# Wrap input in packed sequence, with batch_first=True
packed_input = torch.nn.utils.rnn.pack_padded_sequence(
    batch_in, seq_lengths, batch_first=True)

# Create an RNN object, set batch_first=True
rnn = nn.RNN(input_dim, hidden_size, n_layers, batch_first=True) 

# Run input through RNN 
packed_output, _ = rnn(packed_input)

# Unpack, with batch_first=True.
output, _ = torch.nn.utils.rnn.pad_packed_sequence(
    out, batch_first=True)
print(&quot;Unpacked, padded output: &quot;)
print(output)

# Extract the outputs for the last timestep of each example
idx = (torch.LongTensor(lengths) - 1).view(-1, 1).expand(
    len(lengths), output.size(2))
time_dimension = 1 if batch_first else 0
idx = idx.unsqueeze(time_dimension)
if output.is_cuda:
    idx = idx.cuda(output.data.get_device())
# Shape: (batch_size, rnn_hidden_dim)
last_output = output.gather(
    time_dimension, Variable(idx)).squeeze(time_dimension)
print(&quot;Last output: &quot;)
print(last_output)
</code></pre>
<p>and the output:</p>
<pre><code class="language-dark">Unpacked, padded output:
Variable containing:
(0 ,.,.) =
 -0.0279  0.8709
  0.7806  0.7903
  0.5799  0.9227

(1 ,.,.) =
  0.7244  0.7105
  0.5795  0.8988
  0.0000  0.0000

(2 ,.,.) =
 -0.7699  0.9169
  0.0000  0.0000
  0.0000  0.0000

(3 ,.,.) =
  0.4918  0.4545
  0.0000  0.0000
  0.0000  0.0000
[torch.FloatTensor of size 4x3x2]

Last output:
Variable containing:
 0.5799  0.9227
 0.5795  0.8988
-0.7699  0.9169
 0.4918  0.4545
[torch.FloatTensor of size 4x2]
</code></pre>
<p>As you can see, the code successfully extracted the last-timestep outputs for each example in the batch.</p>
<h2 id="somemorecontextforthosewhomightnotbesuperfamiliarwithpytorch">Some more context for those who might not be super familiar with PyTorch</h2>
<p>PyTorch RNNs return a tuple of <code>(output, h_n)</code>:</p>
<ul>
<li><code>output</code> contains the hidden state of the last RNN layer at the last timestep --- this is usually what you want to pass downstream for sequence prediction tasks.</li>
<li><code>h_n</code> is the hidden state for <code>t=seq_len</code> (for all RNN layers and directions).</li>
</ul>
<p><code>output</code> is a tensor of shape <code>seq_len, batch_size, hidden_size * num_directions</code> if <code>batch_first=False</code> in the RNN, and it&apos;s a tensor of shape <code>batch_size, seq_len, hidden_size * num_directions</code> if <code>batch_first=False</code>.</p>
<p>If you&apos;re using a RNN with variable-length input (made possible with a <code>PackedSequence</code>), <code>seq_len</code> refers to the longest sequence in the <code>PackedSequence</code>. In this case, you often need to extract the output features for each batch at the last timestep (where the &quot;last timestep&quot; is the length of the sequence for the particular example). Note that it doesn&apos;t work to simply use <code>output[-1]</code> (or <code>output[:, -1]</code> if <code>batch_first=True</code>) since outputs beyond an example&apos;s length are padded with 0.</p>
<!--kg-card-end: markdown-->]]></content:encoded></item><item><title><![CDATA[Flattening the Gigaword Corpus]]></title><description><![CDATA[<!--kg-card-begin: markdown--><p><em>Code for flattening the Gigaword corpus and associated usage instructions are at <a href="https://github.com/nelson-liu/flatten_gigaword/">nelson-liu/flatten_gigaword</a></em></p>
<p>The <a href="https://catalog.ldc.upenn.edu/ldc2011t07">English Gigaword Corpus</a> is a massive collection of newswire text; the unzipped corpus is ~26 gigabytes, and there are are ~4 billion tokens. It&apos;s a commonly used corpus for language modeling and</p>]]></description><link>https://blog.nelsonliu.me/2017/09/23/flattening-the-gigaword-corpus/</link><guid isPermaLink="false">5d5962bc2aa85523bca5b520</guid><dc:creator><![CDATA[Nelson Liu]]></dc:creator><pubDate>Sat, 23 Sep 2017 07:39:00 GMT</pubDate><content:encoded><![CDATA[<!--kg-card-begin: markdown--><p><em>Code for flattening the Gigaword corpus and associated usage instructions are at <a href="https://github.com/nelson-liu/flatten_gigaword/">nelson-liu/flatten_gigaword</a></em></p>
<p>The <a href="https://catalog.ldc.upenn.edu/ldc2011t07">English Gigaword Corpus</a> is a massive collection of newswire text; the unzipped corpus is ~26 gigabytes, and there are are ~4 billion tokens. It&apos;s a commonly used corpus for language modeling and other NLP tasks that require large amounts of monolingual English data.</p>
<p>Despite its relative ubiquity, I couldn&apos;t find anything online to do something very simple --- extract the text from all the files in the corpus into one large text file. My motivation for doing this was to train a n-gram language model, but there are a variety of other uses for the flattened data as well.</p>
<h2 id="decompressing">Decompressing</h2>
<p>The Gigaword corpus comes with seven directories of data compressed in <code>gzip</code> format. The first step is to naturally unzip all of it. To recursively unzip all the data in these directories, use the <code>-r</code> flag in <code>gunzip</code>.</p>
<pre><code>gunzip -r /gigaword_path/data/
</code></pre>
<p>If your <code>gunzip</code> doesn&apos;t have this flag, piping the results of <code>find</code> to <code>gunzip</code> should do the trick.</p>
<h2 id="parsingandtokenizinganindividualdatafile">Parsing and tokenizing an individual data file</h2>
<p>In each of the directories, there are a variable number of files. Each of these data files are in SGML format. To parse a single file, I used the <a href="https://www.crummy.com/software/BeautifulSoup/"><code>BeautifulSoup</code></a> library. Extracting the raw text was as simple as finding all the words between <code>&lt;p&gt;</code> tags.</p>
<p>However, after looking at the data I quickly realized that it includes the original linebreaks as found inside the newswire text. Thus, one sentence can often have multiple newlines within it --- this confuses many tokenizers. To deal with this, I replace all consecutive newlines with spaces, and then tokenize each paragraph (block of text in a <code>&lt;p&gt;</code> tag) with <a href="https://spacy.io/"><code>SpaCy</code></a>.</p>
<p>Thus, to parse a file, I:</p>
<ul>
<li>Iterate through all the paragraphs in the SGML</li>
<li>Extract the text, and tokenize it</li>
<li>Write a new flattened file with one paragraph per line.</li>
</ul>
<p>Each line in the output flattened file is thus a paragraph, and the tokens (as delimited by SpaCy) are space-separated. These files are perfectly compatible with language modeling toolkits like <a href="http://kheafield.com/code/kenlm/"><code>KenLM</code></a>.</p>
<p>The script to parse a single file is at <a href="https://github.com/nelson-liu/flatten_gigaword/blob/master/flatten_one_gigaword.py">flatten_one_gigaword.py</a></p>
<h2 id="makingitfastwithparallelprocessing">Making it fast with parallel processing</h2>
<p>Parsing one file can take quite a while (up to around 3 minutes). Combined with the fact that the Gigaword corpus has 1010 files, it&apos;s easy to see how processing the whole dataset be quite slow.</p>
<p>However, the task is embarrasingly parallel, so let&apos;s use multiple cores to flatten files simultaneously and merge them all at the end! This was pretty easily accomplished with <a href="https://www.gnu.org/software/parallel/"><code>GNU parallel</code></a>, like so:</p>
<pre><code>find ${GIGAWORDDIR}/data/*/* | parallel --gnu --progress -j ${NUMJOBS} \
    python flatten_one_gigaword.py \
           --gigaword-path \{\} \
           --output-dir ${OUTPUTDIR}
</code></pre>
<p>This command <code>find</code>s all the data files in the Gigaword directory on the disk, and then runs the <code>flatten_one_gigaword.py</code> file on each of them. The output directory is where the flattened version of each data file is written, and we can simply <code>cat</code> them together at the end to get our desired output. The final output is a file named <code>flattened_gigaword.txt</code> with one paragraph per line and with tokens delimited by spaces.</p>
<pre><code>cat ${OUTPUTDIR}/*.flat &gt; ${OUTPUTDIR}/flattened_gigaword.txt
</code></pre>
<p>The script to parse the entire dataset in parallel is at <a href="https://github.com/nelson-liu/flatten_gigaword/blob/master/flatten_all_gigaword.sh">flatten_all_gigaword.sh</a></p>
<!--kg-card-end: markdown-->]]></content:encoded></item><item><title><![CDATA[Paraphrase Identification Models in Tensorflow]]></title><description><![CDATA[<!--kg-card-begin: markdown--><p>I&apos;ve been loosely hacking on the <a href="https://www.kaggle.com/c/quora-question-pairs">Quora Question Pairs</a> dataset in my free time to get some more experience working with vanilla Tensorflow for NLP in a practical setting. Yesterday, I opened sourced the code I&apos;ve written (with some contributions from <a href="https://github.com/ohkhan">Omar Khan</a>, thanks!) and you</p>]]></description><link>https://blog.nelsonliu.me/2017/05/20/paraphrase-identification-in-tensorflow/</link><guid isPermaLink="false">5d5962bc2aa85523bca5b51b</guid><category><![CDATA[tensorflow]]></category><category><![CDATA[machine learning]]></category><category><![CDATA[open source]]></category><category><![CDATA[python]]></category><category><![CDATA[nlp]]></category><category><![CDATA[paraphrase-identification]]></category><dc:creator><![CDATA[Nelson Liu]]></dc:creator><pubDate>Sat, 20 May 2017 17:44:50 GMT</pubDate><content:encoded><![CDATA[<!--kg-card-begin: markdown--><p>I&apos;ve been loosely hacking on the <a href="https://www.kaggle.com/c/quora-question-pairs">Quora Question Pairs</a> dataset in my free time to get some more experience working with vanilla Tensorflow for NLP in a practical setting. Yesterday, I opened sourced the code I&apos;ve written (with some contributions from <a href="https://github.com/ohkhan">Omar Khan</a>, thanks!) and you can <a href="https://github.com/nelson-liu/paraphrase-id-tensorflow">find it on GitHub (nelson-liu/paraphrase-id-tensorflow)</a>.</p>
<p>To be specific, I implemented the following models in the repo:</p>
<ul>
<li>
<p>A basic Siamese LSTM baseline, loosely based on the model<br>
in<br>
<a href="https://www.semanticscholar.org/paper/Siamese-Recurrent-Architectures-for-Learning-Sente-Mueller-Thyagarajan/6812fb9ef1c2dad497684a9020d8292041a639ff">Mueller, Jonas and Aditya Thyagarajan. &quot;Siamese Recurrent Architectures for Learning Sentence Similarity.&quot; AAAI (2016).</a></p>
</li>
<li>
<p>A Siamese LSTM model with an added &quot;matching layer&quot;, as described<br>
in<br>
<a href="https://www.semanticscholar.org/paper/Learning-Natural-Language-Inference-using-Bidirect-Liu-Sun/f93a0a3e8a3e6001b4482430254595cf737697fa">Liu, Yang et al. &quot;Learning Natural Language Inference using Bidirectional LSTM model and Inner-Attention.&quot; CoRR abs/1605.09090 (2016)</a>.</p>
</li>
<li>
<p>The more-or-less state of the art Bilateral Multi-Perspective Matching model<br>
from<br>
<a href="https://www.semanticscholar.org/paper/Bilateral-Multi-Perspective-Matching-for-Natural-L-Wang-Hamza/b9d220520a5da7d302107aacfe875b8e2977fdbe">Wang, Zhiguo et al. &quot;Bilateral Multi-Perspective Matching for Natural Language Sentences.&quot; CoRR abs/1702.03814 (2017)</a>.</p>
</li>
</ul>
<p>Anecdotally, when I was first starting out with Tensorflow, by far the most effective learning strategy for me was to (1) read a paper, (2) find an open source implementation of it, and then (3) read through the code. Through this, I reinforced my knowledge of both the work described in paper as well as how people write Tensorflow code in practice.</p>
<p>Unfortunately, the largest issue was mostly (2) --- it&apos;s hard to find well-written projects that explain what&apos;s going on with minimal magic involved. This is fairly understandable, though, as research code is basically written ... for the sole purpose of research. Other people are rarely going to run your code after the fact, and frankly researchers get a gold star just for open-sourcing anything regardless of quality; there&apos;s thus little incentive to actually write your code as if other people would use it.</p>
<p>To this end, I&apos;ve took painstaking care to try to implement the models &quot;right way&quot; by adhering Tensorflow best practices, document the code (through function docstrings and comments in general), and provide tests (98% coverage + running on a CI server).</p>
<p>Hopefully someone finds this helpful, and I&apos;m happy to answer questions about the project and/or provide more info about how to get started with Tensorflow.</p>
<!--kg-card-end: markdown-->]]></content:encoded></item><item><title><![CDATA[Installing and Updating GTX 1080 Ti Drivers / CUDA on Ubuntu]]></title><description><![CDATA[<!--kg-card-begin: markdown--><p>I recently had to figure out how to set up a new Ubuntu 16.04 machine with NVIDIA&apos;s new GTX 1080 Ti graphics card for use with CUDA-enabled machine learning libraries, e.g. Tensorflow and PyTorch; since the card (as of this writing) is relatively new, the process</p>]]></description><link>https://blog.nelsonliu.me/2017/04/30/installing-and-updating-gtx-1080-ti-cuda-drivers-on-ubuntu/</link><guid isPermaLink="false">5d5962bc2aa85523bca5b518</guid><category><![CDATA[machine learning]]></category><category><![CDATA[python]]></category><category><![CDATA[nvidia]]></category><category><![CDATA[CUDA]]></category><category><![CDATA[drivers]]></category><category><![CDATA[tensorflow]]></category><dc:creator><![CDATA[Nelson Liu]]></dc:creator><pubDate>Sun, 30 Apr 2017 00:41:01 GMT</pubDate><content:encoded><![CDATA[<!--kg-card-begin: markdown--><p>I recently had to figure out how to set up a new Ubuntu 16.04 machine with NVIDIA&apos;s new GTX 1080 Ti graphics card for use with CUDA-enabled machine learning libraries, e.g. Tensorflow and PyTorch; since the card (as of this writing) is relatively new, the process was pretty involved. The same tricks should also work for the newer Titan Xp graphics card.</p>
<p>Edits<br>
(02/01/2019):<br>
I&apos;ve updated the install instructions to use driver version 410 (necessary for CUDA 10, but should retain backwards compatibility with older CUDA versions).</p>
<p>(1/27/2018):<br>
Tensorflow 1.5.0 and PyTorch 0.3 now have pre-built binaries for CUDA 9. <strong>If you install CUDA 9, the driver version that comes with it should be fully compatible with the 1080 Ti</strong>. You can easily install CUDA 9 on most Linux distributions with your package manager (<a href="http://docs.nvidia.com/cuda/cuda-installation-guide-linux/#package-manager-installation">see here for details</a>).</p>
<p>If you want to use CUDA 8 for some reason (e.g. using an older Tensorflow), read on...</p>
<p>(5/10/2017):<br>
Looks like driver version 381 is out of beta and on the PPA, so I&apos;ve updated the recommended driver versions and install instructions accordingly.</p>
<h3 id="1installcudawithoutthedriver">1. Install CUDA without the driver</h3>
<p>I couldn&apos;t just install CUDA and have it work, since certain CUDA version (e.g., 8.0) come with a driver version (in the case of CUDA 8.0, driver version <code>375.26</code>) that doesn&apos;t support the GTX 1080 Ti and other newer cards. As a result, installing CUDA from <code>apt-get</code> doesn&apos;t work since it installs this driver version. Thus, <strong>you have to <a href="http://docs.nvidia.com/cuda/cuda-installation-guide-linux/#runfile">install with the runfile</a>, to opt-out of installing the driver.</strong></p>
<p>When running the installer, make sure to not install the driver that comes with CUDA. We&apos;ll install the driver with <code>apt-get</code> in the next step.</p>
<p><strong>Post Install Notes</strong> (Thanks to Jake Boggan for mentioning this in the comments): After installing, check that the CUDA folders are where you expect them to be (usually <code>/usr/local</code>). The CUDA installer creates a symlink at <code>/usr/local/cuda</code> that automatically points to the version of CUDA installed.</p>
<p>Make sure to add <code>/usr/local/cuda/bin</code> to your <code>$PATH</code>, and <code>/usr/local/cuda/lib64</code> to your <code>$LD_LIBRARY_PATH</code> if you&apos;re on a 64-bit machine / <code>/usr/local/cuda/lib</code> to your <code>$LD_LIBRARY_PATH</code> if you&apos;re on a 32-bit machine. There&apos;s a bit more info at the <a href="http://docs.nvidia.com/cuda/cuda-installation-guide-linux/#post-installation-actions">CUDA docs</a>, but the paths will likely differ based on version so be sure to manually verify that the folders you&apos;re adding to the environment variables exist.</p>
<h3 id="2installingthedriverwithaptget">2. Installing the driver with apt-get</h3>
<p>To install the driver with <code>apt-get</code>, I used the Ubuntu <a href="https://launchpad.net/~graphics-drivers/+archive/ubuntu/ppa">graphics-drivers PPA</a>. This method isn&apos;t officially supported by NVIDIA, but it seems to work well for many people.</p>
<p>At the <a href="https://launchpad.net/~graphics-drivers/+archive/ubuntu/ppa">graphics-drivers PPA homepage</a>, there&apos;s a listing of the various graphics drivers that they offer; check the <a href="http://www.nvidia.com/Download/index.aspx">NVIDIA download website</a> to figure out what version of the driver you need for your card. If it&apos;s in the PPA, great! If not, you unfortunately have to wait for them to add it. They&apos;re pretty timely, though.</p>
<p>Add the PPA to <code>apt-get</code> and update the index by running:</p>
<pre><code class="language-language-bash">sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt-get update
</code></pre>
<p>Now, we use it to install the desired driver versions (Major version <code>410</code> as of this writing):</p>
<pre><code class="language-language-bash">sudo apt-get install nvidia-410
</code></pre>
<p>Reboot your computer, and the GPU should run on the new driver. To verify, run <code>nvidia-smi</code> and confirm that the <code>Driver Version</code> at the top of the output is what you expect and that the rest of the information looks good.</p>
<p>You should now be able to fire up Python and <a href="https://www.tensorflow.org/tutorials/using_gpu#logging_device_placement">test that it works with Tensorflow</a> or your favorite deep learning framework.</p>
<h3 id="3verifyingtheinstallationworked">3. Verifying the installation worked</h3>
<h4 id="cuda">CUDA</h4>
<p>To test the CUDA installation, you can run the <code>deviceQuery</code> example bundled with CUDA. If you navigate to the CUDA samples folder (<code>/usr/local/cuda#.#/samples</code> or <code>~/NVIDIA_CUDA-#.#_Samples</code> by default), you can find the <code>deviceQuery</code> example in <code>&lt;samples_dir&gt;/1_Utilities/deviceQuery</code>.</p>
<p>Running <code>make</code> in this directory should compile the CUDA source file to produce a binary that will produce a variety of statistics about your GPU and run some test on it. Run the binary with <code>./deviceQuery</code>, and you should see a bunch of output about your device; here&apos;s <a href="https://gist.github.com/nelson-liu/623eb54d977c98db005eaf2fbc449238">my output with a 1080 Ti for comparison</a>.</p>
<h4 id="drivers">Drivers</h4>
<p>If the driver installation went properly, you should be able to run <code>nvidia-smi</code> and get an output like the one below (the memory usage / temp / fan / GPU utilization will probably differ, since this was measured under load). Make sure that the version displayed in the top-left corner is the same as the one you expect:</p>
<pre><code>Sun May  7 19:54:19 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 381.09                 Driver Version: 381.09                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 0000:02:00.0     Off |                  N/A |
| 42%   73C    P2   194W / 250W |   8417MiB / 11172MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
</code></pre>
<p>When I was using driver version 378, it oddly didn&apos;t show the name as <code>GeForce GTX 108...</code>, but rather as just <code>Graphics Device</code>. The card worked fine with TensorFlow, though.</p>
<p>It&apos;d probably be a good idea to test that your GPU works with your machine learning library of choice, <a href="https://www.tensorflow.org/tutorials/using_gpu#logging_device_placement">here are instructions for doing so on Tensorflow</a>.</p>
<h3 id="forthefutureupdatingtheaptgetdrivers">For the future: updating the apt-get drivers</h3>
<p>It&apos;s pretty easy to upgrade the drivers to a different version.</p>
<p>First, remove the old drivers:</p>
<pre><code>sudo apt-get purge nvidia*
</code></pre>
<p>Now, just install the new driver with the PPA as detailed above and reboot.</p>
<!--kg-card-end: markdown-->]]></content:encoded></item><item><title><![CDATA[Making autoenv + conda faster]]></title><description><![CDATA[<!--kg-card-begin: markdown--><p>I&apos;ve recently switched over to using the fantastic <a href="https://github.com/kennethreitz/autoenv">autoenv</a> to automatically activate my anaconda environments and set necessary environment variables when I enter a directory on my terminal. You basically write some bash code in a <code>.env</code> file, put it into a directory, and autoenv will automatically run</p>]]></description><link>https://blog.nelsonliu.me/2017/03/26/making-autoenv-conda-faster/</link><guid isPermaLink="false">5d5962bc2aa85523bca5b517</guid><category><![CDATA[python]]></category><category><![CDATA[conda]]></category><category><![CDATA[tooling]]></category><dc:creator><![CDATA[Nelson Liu]]></dc:creator><pubDate>Sun, 26 Mar 2017 21:51:32 GMT</pubDate><content:encoded><![CDATA[<!--kg-card-begin: markdown--><p>I&apos;ve recently switched over to using the fantastic <a href="https://github.com/kennethreitz/autoenv">autoenv</a> to automatically activate my anaconda environments and set necessary environment variables when I enter a directory on my terminal. You basically write some bash code in a <code>.env</code> file, put it into a directory, and autoenv will automatically run <code>.env</code> when you enter the directory or any of its subdirectories.</p>
<p>However, I found that putting just <code>source activate desired_environment</code> in my <code>.env</code> (to activate the <code>desired_environment</code> conda environment) made my shell <em>very</em> slow --- I&apos;d have to wait ~2 seconds after issuing a <code>cd</code> into a directory with a <code>.env</code> file (or a subdirectory of one).</p>
<p>The following bash snippet makes activating conda environments with <code>autoenv</code> a lot faster:</p>
<pre><code class="language-language-bash">current_environment=&quot;&quot;
environment_to_activate=test

# $CONDA_PREFIX is non-empty when in an environment
if [[ $CONDA_PREFIX != &quot;&quot; ]]; then
  # Get the name of the environment from the path
  current_environment=&quot;${CONDA_PREFIX##*/}&quot;
fi

if [[ $current_environment != $environment_to_activate ]]; then
  # We are not in the environment to activate, so activate it.
  source activate $environment_to_activate
fi
</code></pre>
<p>The snippet basically checks if you&apos;re already in the conda environment you want to activate (called <code>test</code> in this case, and assigned to <code>environment_to_activate</code>), and doesn&apos;t rerun the slow <code>activate</code> script if you are. Handy!</p>
<p>To use this snippet, just drop it into your <code>.env</code> file and replace <code>test</code> with the name of whatever environment you want to activate; your shell should feel a lot less slow.</p>
<!--kg-card-end: markdown-->]]></content:encoded></item><item><title><![CDATA[Easy Progress Bars For Python File Reading with tqdm]]></title><description><![CDATA[I've been a fan of the tqdm Python module for quite some time, but I found it difficult to find a reason to use it.]]></description><link>https://blog.nelsonliu.me/2016/07/30/progress-bars-for-python-file-reading-with-tqdm/</link><guid isPermaLink="false">5d5962bc2aa85523bca5b511</guid><category><![CDATA[python]]></category><dc:creator><![CDATA[Nelson Liu]]></dc:creator><pubDate>Sat, 30 Jul 2016 03:47:00 GMT</pubDate><content:encoded><![CDATA[<!--kg-card-begin: markdown--><p>I&apos;ve been a fan of the <a href="https://github.com/noamraph/tqdm"><code>tqdm</code></a> Python module for quite some time, but I found it difficult to find a reason to use it; generally, loops run fast enough that a progress bar is unnecessary. However, I found a perfect use for it in reading large files.</p>
<p>If the task isn&apos;t something I can speed up via <code>multiprocessing</code>, I can use <code>tqdm</code> to decide whether I can grab a cup of coffee or work on something else while I let it run. <code>tqdm</code> allows me to easily add a progress bar to the read operation, like so:</p>
<pre><code class="language-python">with open(file_path) as file:
    for line in tqdm(file, total=get_num_lines(file_path)):
        # various operations here
</code></pre>
<p>As you can see, adding this functionality is as simple as wrapping the file with the <code>tqdm</code> method. However, to display the progress bar, <code>tqdm</code> needs an idea of how many total lines it needs to process. I use <a href="http://stackoverflow.com/a/850962/1877942">this code snippet</a> from StackOverflow to quickly find this information when instantiating the progress bar:</p>
<pre><code class="language-python">import mmap

def get_num_lines(file_path):
    fp = open(file_path, &quot;r+&quot;)
    buf = mmap.mmap(fp.fileno(), 0)
    lines = 0
    while buf.readline():
        lines += 1
    return lines
</code></pre>
<p>Here&apos;s what it looks like in action:<br>
<video width="702" height="514" autoplay loop poster="http://thumbs.gfycat.com/ElasticSameAmericanrobin-poster.jpg"><br>
<source src="http://zippy.gfycat.com/ElasticSameAmericanrobin.webm" type="video/webm"><br>
<source src="http://zippy.gfycat.com/ElasticSameAmericanrobin.mp4" type="video/mp4"><br>
</video></p>
<p>Pretty neat, in my opinion!</p>
<p>If you have any questions, comments, or suggestions, you&apos;re welcome to leave a comment below.</p>
<!--kg-card-end: markdown-->]]></content:encoded></item></channel></rss>