30 Days in the Life of a Machine Learning Researcher

As a PhD student, you’ll find it hard not to think about research problems even when you’re in one of the most beautiful places on Earth.

January 2: Leaving Family and Returning to Stanford

I had spent most of winter break with my family in Arkansas, but on the 2nd of the month, I returned to Stanford. Technically, there was another week of vacation listed on Stanford’s academic calendar, but PhD students do not follow the typical academic schedule — any breaks are decided between you and your advisor.

January 3–4: Implementing a VAE for Paper 1

My first order of business was to clearly define the problem that I had been working on in bits and pieces before winter break, as well as code up a baseline that I could use to compare to my proposed algorithm.

Here, a VAE trained on celebrity images is used to generate new celebrity images, by varying one dimension at a time. The dimension going from bottom-left to top-right seems to represent hair length (or background color), while the dimension going from top-left to bottom-right seems to represent skin color. Since we only allow 2 latent variables, the generated images are not that realistic.
  • Downloading and preprocessing the CelebA image dataset
  • Writing the code for a VAE in TensorFlow (Keras), and training the VAE on the CelebA image dataset.

Jan 5–6: Working Through the Weekend

I’ve told myself many times that I will not work on Saturdays and Sundays, and that I need to discover a healtheir work-life balance. But when the going gets tough, I immediately default to hitting the lab on weekends. Realizing that I had about three weeks until ICML, and with a meeting with my advisor coming up on Tuesday, I bit through my frustration and spent most of the weekend in my office, skipping a ski trip that my friends planned to Tahoe.

Examples of images in the target dataset (left) and background dataset (right).

I’ve told myself many times that I will not work on Saturdays and Sundays… but when the going gets tough, I immediately default to going back to the lab on weekends.

Jan 7, 2019: Troubleshooting our GPU Cluster, Babbage 3

The next morning, I came back to find that my simulation hadn’t produced great results — in fact, it hadn’t produced results at all! Shortly after I had started the simulation, the lab cluster that we use to run our simulations, Babbage 3, had crashed, leaving me staring at the stack trace:

ResourceExhaustedError: OOM when allocating tensor

Jan 8: Meeting with my Advisor, Richard

By the next morning, I finally had some results using the contrastive VAE architecture on the celebrity image dataset. With generative models, evaluating the results can be tricky, because image quality and latent feature selection can be subjective. It seemed to me that the results were not that great, but were a definite improvement over the standard VAE. I quickly copied and pasted the figures into a set of slides so that I would have them ready to present to my advisor, Richard.

Gene expression matices often show high degrees of correlation. Do we really need to measure every genes, or can we just measure a few (saving experimental costs and time) and then impute the rest?
Figure showing the methodology of the contrastive VAE paper. The early version is shown on the left, while the final version, which appeared in the paper, is shown on the right.
Simplified architecture of the concrete autoencoder. The figure is generated entirely from LaTeX.

“A reader should understand your paper just from looking at the figures, or without looking at the figures” — Zachary Lipton

Jan 10: Working with an Undergraduate Student on Paper 2

(a) Figure showing the 20 most important pixels (out of a total of 784) (b) Sample images from the MNIST dataset (c) Showing the 20 pixels that were selected from each sample image in the previous panel. (d) The reconstructed images using only the 20 selected pixels, which approximate the original images quite well. Thank you Melih for making this figure and not getting frustrated by my comments!
Figure adapted from Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
Here, a contrastive VAE trained on celebrity images with hats (target) and without hats (background) is used to generate new celebrity images, by varying one dimension at a time. The vertical dimension seems to involve hat color, while the horizontal dimension seems to involve hat shape. Since we only allow 2 latent variables, the generated images are not that realistic.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store