Deep neural networks can be thought of as compositions of many simple transformations, each represented by a layer with trainable parameters. When the number of layers is large, the effect of multiplying many random matrices becomes exponentially unstable, i.e. they can grow or shrink exponentially. This is the primary reason that naive initialization leads to exploding or vanishing signals for both forward (activations) and backward (gradients). Nonetheless, stability is possible when each layer is close to the identity operation. With the right scaling of weights at initialization, a deep network acts like a time-discretized flow, and the total transformation resembles a matrix exponential of small perturbations. [...]
AI is here, and its impacts on education cannot be overstated. Let’s put aside the issues of cheating; I assume that you want to learn, perhaps with the assistance of LLMs if they are actually helpful. But how do you know you’re not using AI as a crutch, versus using it to augment learning? The former setting outsources your thinking to AI, whereas the latter can help you reveal gaps in your understanding, bypass blockers that prevent learning, and/or tailor education to your style. In this post, I provide an analogy between learning and phase transitions in statistical mechanics, and describe recommendations and warnings on using LLMs in different learning scenarios. [...]
With the Euclid and Roman Space Telescope missions ready to image billions of galaxies, we’ll need data-driven methods to find new, rare phenomena that exist outside human-defined taxonomies! Sparse Autoencoders (SAEs) can be that discovery engine, surfacing interpretable features in modern galaxy surveys. This blog post highlights some preliminary results from our tiny NeurIPS ML4PS workshop paper, jointly led by Mike Walmsley and me. Read the paper here. [...]
This post discusses how graph neural networks (GNNs) can model the galaxy–halo connection within its large-scale surroundings. Dark matter structures, which seem to account for most of the mass in the Universe, can be represented as nodes in a cosmic graph. But dark matter—which solely interacts via gravitation—is also much easier to simulate than the messy baryons, whose magnetohydrodynamics are computationally expensive. By exploiting the representational power of GNNs, can we predict galaxies’ baryonic properties purely using simple dark matter-only simulations? Yes we can! [...]
The Eleven Laws of Showrunning by Javier Grillo-Marxuach is full of useful advice for management and operations. Nominally, it’s about how to deliver a television show, from ideation to writing to production to postproduction, but there’s a ton of guidance that’s surprisingly relevant for working with large language models (LLMs). [...]
In the book Impro: Improvisation and the Theatre, Keith Johnstone recounts a moment between a teacher and a special needs student. The teacher holds up a flower and says, “Look at the pretty flower.” The girl responds, “All of the flowers are beautiful.” Then the teacher gently says, “but this flower is especially beautiful.” The girl proceeds to scream and thrash about violently. [...]
Many physical phenomena exhibit relational inductive biases and can be represented as mathematical graphs. In recent years, graph neural networks (GNNs) have been successfully used to model and learn from astronomical data. This post provides an introductory review to GNNs for astrophysics. [...]
If you’re a blogger or researcher sharing your work online, you’ve probably wondered: is social media actually useful for disseminating your writing? I’ve been asking myself this question since returning to blogging just over a month ago. [...]
I am a self-confessed productivity junkie. I hate wasting time. And if you scroll through social media, or even my blog posts, you might think that the typical research or learning process is just a happy, monotonic hill climb, capped off with regular announcements of new discoveries or gained expertise. But what if the most important lessons emerge not from unencumbered progress, but rather from seemingly aimless pursuits and the frustration of doing things badly? This post is a tribute to all those times we got stuck and emerged with nothing to show for it, because those “unproductive” moments lead to some of the most important lessons we can ever learn. [...]
Here’s a casual introduction to foundation models and how they might impact astronomy research in the coming years. I’m writing this on the train back from New York to Baltimore, having just wrapped up the Foundation Models in Astronomy workshop at the Flatiron Institute Center for Computational Astrophysics. My co-organizers and I are planning to write up a more comprehensive blog post based on our workshop discussions; in the meantime, you’ll just have to settle for this. [...]
Need to perform a boring, repetitive task? Even if it can’t be fully automated, you may be able to dramatically speed up your task by partially automating it! Simply use a LLM to code up a throwaway app to help accelerate your mindless task. [...]
Back in 2014, I was privileged to participate in the Vatican Observatory Summer School (VOSS). Over those four weeks, I formed new friends, made new discoveries, and ate awesome food. But the most unforgettable moment of that trip was meeting Pope Francis. [...]
To truly know how well a machine learning model performs, you need a reliable evaluation set. This post explains a practical way to create such a high-quality dataset, often called a golden sample, and use it to compute unbiased evaluation metrics. [...]
Large language models (LLMs) haven’t upped my productivity by 10x, but they have dramatically changed the way that I work. In this post I introduce four ways that I use LLMs every day. [...]
Writing these posts is fun but time-consuming. How can I stay motivated enough to post consistently? To sustain a habit of writing, I’ll need to create an easier path for my future self.[...]
In the previous post, we examined the feature space of galaxy morphological features. Now, we will use the Grad-CAM algorithm to visualize the parts of a galaxy image that are most strongly associated with certain classifications. This will allows us to identify exactly which morphological features are correlated with low- and high-metallicity predictions. [...]
Let’s explore the morphological feature space of galaxies represented by a trained CNN. We will use PCA to reduce the dimensionality of the neural network’s latent features, and then visualize these features with matplotlib. [...]
We are becoming more and more reliant on machine learning algorithms in our everyday lives. But what if these algorithms aren’t fair? In this exploratory data analysis of Obermeyer et al. 2019, we look at how racial biases can creep in to an algorithm’s decision-making process. [...]
Let’s train a deep neural network from scratch! In this post, I provide a demonstration of how to optimize a model in order to predict galaxy metallicities using images, and I discuss some tricks for speeding up training and obtaining better results. [...]
Welcome! In this first post to my blog, we will take a deeper look at galaxy images. Why should we bother measuring the metallicities, or elemental abundances, of other galaxies? And why would we use convolutional neural networks? Read more to find out! [...]
