S

Samarth's Notes

I am a journalist and a programmer. I share raw thoughts, ideas and experiences on this blog. For occasional updates about my work, sign up here: bansalsamarth.substack.com

#11: Lessons learnt from my 52-week workout streak

Today, I have hit a personal milestone: cult.fit app informs me I have been regular with my workouts for 52 weeks in a row — meaning, over the last year, I exercised every single week without exception. From being a high school student who preferred to bunk the sports period to go study in the library, I have come a long way.

Yes, there were bad weeks where my intensity was low or I could pack in just one session. But they were few in number. I don't have the exact stats, but I estimate my hit rate was ~4-5 sessions per week, focused on strength training and cardio, skewed towards the former. Irrespective of the metrics, fitness is primarily about being consistent—and I am glad that my timeline has stretched to one full-year.

Blogging today to document a few thoughts and lessons learnt in this process:

1. How it started: My fitness journey began with an ambitious fat loss goal. I had to maintain a calorie deficit and the primary point of control was food: you can not out-train a bad diet. But of course, workouts are absolutely crucial. Two reasons.

First, to burn more calories and increase my daily energy expenditure (which helps in maintaining deficit). And second, to ensure my weight loss is driven by fat loss and not muscle loss: strength training is crucial to optimise for that goal.

And so I did. It took me seven months — July 2020 to January 2021 — to reach my target of losing 20kgs. (I will blog about my fat loss journey in great detail in a future post.)

Those seven months completely changed my outlook about workouts: no more did it feel like a chore, something I had to forcefully do just because doctors says it's good. I experienced what regular exercise did to me: high energy levels through the day, more strength, better mood and increased capacity to move around. Most importantly: I feel so good after every session — that in itself makes the effort worth it. It is, as some people call it, free therapy.

2. Automating my behaviour: Will power and motivation are overrated. Follow as many fitness instructors on Instagram; watch as many YouTube talks explaining why exercise matters; read any number of books on health; enrol in however expensive fitness programs — none of that works in the long term if you don't convert your motivation to be fitter into a habit operating on auto-pilot. How to do that?

Here is an excellent idea I learnt from behavioural economist Sendhil Mullainathan. In an episode of the Knowledge Project podcast, he said: going to the gym everyday is much easier than going three days a week. It removes a really difficult hurdle in life: choosing. To install the habit, just go every day — even on days you are pressed for time. Go and leave in five minutes. Showing up is the victory, Mullainathan says. Keeping the bar low and staying consistent is what makes desirable habits stick.

Mullainathan makes the case for eliminating decisions and following a pre-decided list of rules:

The rule versus decision has another element which I really like, which is, there’s a person in my life I just do not trust. Even though I spend a lot of time with this person, I just do not trust them. It’s embarrassing to say, but I don’t trust their motives, I don’t trust them to follow through on anything. And that person is me.

You should not trust yourself? So decisions, it’s like handing over bubblewrap to a five year old. You know what’s going to happen. So rules versus decisions are in part also the recognition that you have an uneasy relationship with yourself.

I relate to this so much. Decision making can be especially hard in the overwhelming world of fitness. As Daniel Lieberman writes in his book, most people are constantly exercised about exercise: the commodification, commercialisation and industrialisation of exercise rarely make it fun. Rampant confusion and doubts only make things worse. Gyms and coaches make a hell lot of money feeding on people's anxieties and insecurities about their bodies. It is easy to get stuck in a decision paralysis, constantly doubting you are missing out and not doing the right thing, not enjoying the process and falling off the track.

If you just simplify what you intend to do and stay committed to it, you are more likely to stay on course. I was able to do that thanks to one of my favourite Indian startups: cult.fit. The Bengaluru-based startup has reduced the friction in accessing fitness: their online product for at-home workouts was just what I needed to meet my goals. They make fitness fun: the expansive variety ensures that the workouts are not monotonous, and I remain excited about what's coming in the next session. 

Cult reduced the number of decisions I had to make: I only had to decide what my weekly workout mix needs to look like and then, as a rule, go to the app and do the session. It's that simple.

3. A strategy that helped me inculcate exercise in my daily routine: doing it first thing in the morning. This works brilliantly for me. Random things happen as the day progresses: an unscheduled phone call, an unexpected assignment at work, mood swing and what not. The result: the 6pm exercise plan falls apart. Mornings are less random, and the likelihood of meeting your goal is higher.

Morning workouts have two other advantages. One, I feel a sense of accomplishment, having struck off one task from my todo list early in the morning. Second, I start my workday with higher energy levels.

4. Finding the time: I am convinced that being busy is not a good excuse for missing out on my workouts. If it features on my priority list, I am able to find time. When I try to bluff myself, I think about Barack Obama's routine. In his memoir published last year, the former President of the United States told us about his workout schedule (Pg 290):

"Each morning, Monday through Thursday, Michelle and I began our days with both Cornell [his athletic trainer] and Sam [his chef], the four of us gathering in the small gym on the third floor of the residence...For both Michelle and me, that daily hour in the gym became one more zone of normalcy...."

If Obama could find time for working out when he had one of the most busiest and stressful jobs in the world, I can too.

5. Mindset: Do it long enough and fitness becomes a part of your identity. Daily choices start revolving around it. This doesn't go unnoticed and you experience a less-talked about problem: fit-shaming. You are shamed for your health choices (those judgemental eyes, gosh!). Multiple fitness-focused friends have experienced this nuisance.  

You eventually learn to ignore it. But I find it useful to keep the big picture in mind and ask the most basic question: why am I even doing this? 

Which brings me to the distinction between two wildly different motivations for staying fit: one is driven by aesthetics; the other keeps good health at the center. While the path to achieve both ends is the same, the reason to prioritise fitness (whether for external validation or internal satisfaction) may have a huge impact on how you feel about your workouts and its outcomes.

Staying healthy is my primary driver. I am not saying aesthetics don't matter to me: I feel good when I look good, or when the shirt I like fits me well, or when a cute girl in the coffee shop cares to notice my existence. But getting three extra right swipes on a dating app can not be the reason to get off my bed at 5.30am to lift dumbbells.

How I look has no direct impact on what I do on a day to day basis: I am a knowledge worker and I get paid for reporting, writing, thinking and coding. If I had a lot of extra time, I would spend it to sharpen my mind. Not only does exercising my mind matter for my professional growth, it is also the thing I enjoy the most: I have been a specs-wearing nerd since teenage and losing a few inches around the waist and adding some around the biceps is not going to change how I think about myself.

In that sense, a commitment to fitness is a philosophical choice, recognising that I can not fully cherish the gift of life and enjoy things I love without good health. It is a commitment to treat the body I occupy with the respect it deserves. 

This is the push that made this one-year streak possible. All other outcomes are amazing and great, but they remain secondary.

#10: Why I am relearning statistics

I have finally started what I had been thinking of doing for at least two years now: relearning statistics. I am a freelance journalist/programmer, and I control the volume of work I want to hold at any point. To make time for this academic pursuit, I consciously reduced my workload—it's a struggle to say no to exciting journalism assignments, but I try.

I am six weeks into this learning endeavour and it's truly rewarding: I am having a lot of fun diving into mathematical theory and solving problems for hours at stretch. If things go as per the plan, I should be doing this till the end of this year—I hope to stay on course.

In a series of blog posts, I will document my learning approach, my renewed appreciation for MOOCs and thoughts on higher education and pedagogy. I am starting today by documenting why I am doing this.

The goal is pretty clear: learning probability theory and statistics from the ground up, building strong foundational understanding, and getting my hands dirty with the underlying math — just like a good mathematics undergraduate student. The irony is that I have an undergraduate degree in mathematics and I did not do what I am doing now back then.

I did have a compulsory course in probability and statistics in my curriculum but I did not take the subject matter seriously (you can pass the course—without an honourable grade—without really understanding the material). I ignored it despite being quite excited about the prospects of data science, simply because I was not aware of the centrality of probability theory in data analysis. I spent a lot of time learning about databases, cloud computing, big data systems, machine learning algorithms, writing code to solve problems and other stuff to get better at all data-related stuff. My focus leaned towards computing and less on core statistics.

It is deeply embarrassing to acknowledge my ignorance of basic statistical ideas that I did not know at the time I proudly graduated with a degree in mathematics and computing from one of the finest universities in India.

But none of this created any problems in my career: I got a break in journalism because of my data skills. To be fair to myself and not be completely self deprecating, I do a decent job with numbers — and that shows up in the body of journalistic work I have produced (more on data journalism in an earlier blog post). I have a data-driven lens to look at the world and an empirical approach is rooted in my thinking. On the practical side, I routinely write code to scrape and clean data, find trends to inform readers, and visualise them for easy interpretation.

Meanwhile, few things happened:

1. Exposure to the distinction between descriptive and inferential statistics

Descriptive part is easy. It largely deals with summary statistics (like mean and median) and tells us what the data is telling about the sample we are looking at. But I figured I was clueless about inference, which, simply put, is the process of drawing conclusions about a population based on a sample or subset of the data.

Yes, you can draw meaningful insights from basic exploratory analysis and computation of summary stats, but inference uses probability theory to substantiate those conclusions and quantify the uncertainty that comes with sample estimates.

For example:

a) I can analyse the state of the economy by putting together datasets of key indicators (like GDP and unemployment) and analyse what it's telling us. That's just simple data interpretation. What I don't understand are the statistical processes used to estimate those indicators: the method to calculate the GDP or unemployment numbers.

As a result, when the policy debate on India's revised GDP series was dominating the news cycle, I was unable to rigorously think about it: it was about the methodology. Similarly, when the Indian government changed the way it conducts the labour force survey (which is the basis of unemployment figures), I was able to follow the political debate; not the statistical one.

b) If I am told that the unemployment went up from 4.2% to 4.9% — how do I assess if this rise is statistically significant or not? What if the change is simply a reflection of random fluctuation? How should I think about the threshold? At the moment, I can't answer these questions with mathematical precision.

c) I can scan empirical studies and research papers and ask questions about the results. But I can't meaningfully engage with the processes used to arrive at the conclusions. Statistical inference lies at the heart of scientific inquiry and not knowing its fundamentals means I can not fully participate in understanding what's really going on. One can argue that you don't necessarily need to know the specifics of methods all the time—and I agree. I am merely suggesting that I am not confident in my arguments without understanding the foundations — and I want to fix that.

These are just a few simple examples that I could immediately think of. I am not even getting into the possibilities of statistical modelling that remain out of my reach — and I have encountered practical problems where my analysis would have been much richer had I been equipped with those tools.

Simply put, the missing bit in my statistical understanding had been hampering my ability to think about questions I care about.

2. Valuing the importance of statistical theory: Here is a paragraph from the book All of Statistics by Larry Wasserman (which I am using as a reference textbook) that puts it clearly.

Students who analyze data, or who aspire to develop new methods for analyzing data, should be well grounded in basic probability and mathematical statistics. Using fancy tools like neural nets, boosting and support vector machines without understanding basic statistics is like doing brain surgery before knowing how to use band-aid.

On point: if only I had this perspective during my undergrad, I might have made different choices. It also points to my failure of appropriately managing a mix of theory and application. Both matter, but my choices and interests were heavily skewed towards application, coding and problem solving, which came at the cost of downplaying the importance of theoretical ideas.

3. A probabilistic worldview: The way I look at the world completely changed once I started appreciating the role of chance in everyday life. The world is not deterministic — it is fundamentally probabilistic. And we underestimate how much it actually matters.

For instance, the (frequentist) probabilistic worldview tells you that if we were to re-run the events of history 100 times, 100 different realities will emerge. The one we are living right now was not "meant to be" — it is just one of the infinite paths the world could have taken. This raises complicated cause-effect questions ("How would things be had the low-probability event X hadn't occurred?")

This thinking emerged primarily from reading. I had read Leonard Mlodinow's The Drunkard's Walk: How Randomness Rules Our Lives a decade ago (sometime in 12th grade); but it was only after I graduated from university and read the work of Nassim Nicholas Taleb, Philip Tetlock (and a few others) that ideas related to risk and uncertainty got baked in my head.

And then I hit a roadblock: how do I transfer my intuitive ideas about probability to practical problem solving? Learning that is one of my goals.

4. Journalism and epistemology: The primary goal of journalism is truth-seeking (this is what I think about my profession; others may propose different end goals: eg social impact, political influence, empowering individuals).

Which is why I think a lot about epistemology—the branch of philosophy that deals with the thorny question of "knowing": how do we know what we know?

A fundamental problem with many journalists is their confidence that they know things and can explain it well, but it should be news to no one that most of it is just garbage. Just look at the deluge of shallow political theories dumped upon us after every election, only to change in the next — it is so frustrating. As Tetlock writes in his book Superforecasting, the problem is that we move too fast from confusion and uncertainty to a clear and confident conclusion without spending enough time in between.

As I wonder about epistemic methods, a science-based statistical inquiry dominates everything else. But my own work told me this approach has its limitations and that I need to expand my horizon. I want to stress test and think more deeply about the limitations of statistics and the power of possible alternatives.

American Historian Jill Lepore stated this brilliantly in a panel discussion:

Even journalism now, is like, "Oh, we should read the 538 because that's data" and everything else is just opinion. We have a kind of cultural worship of data. The bigger question is what crap you can get away with now by saying you're working with data, and what you can impose on other people, and even foot a tax bill for by saying that.

It diminishes all other ways of knowing in realms of knowledge. That is a huge crisis we can't understand. You might know more about the crime I committed by reading a poem, than working with this algorithm. But we don't think about that poem as a form of knowledge. The ways in which this, the reign of data, discredits all kinds of realms of knowledge, and among other things, the experience of women and children and families and the intimate and the sexual and demeans the private as something that can purely just exist for commodification.

There's a whole set of assumptions in that world that we should be talking about. I mean not to say there's no amazing extraordinary research being done that is data-driven or that falls under the heading of data science, but there were a lot of mistakes made when people decided in the 1890s that social science would solve every problem. It was kind of important for other people to say, "You know what, social science can't necessarily solve every problem. It's really useful, but it's important to think about when we should use it and when not."

Has there ever been a time when we have said that what we're doing doesn't solve every problem?

I am hoping that stronger fundamentals will allow me to proportionally place statistics in the broad scope of epistemic methods, displace it where it should not be and make space for alternatives which I continue to explore.

I am excited to see how my learning process evolves. I don't expect to get answers to all of my questions right after completing the coursework I am pursuing (more on that in a future blog post) but I find it helpful to sketch out the big picture as you mess around with minute details. If nothing else, it's a good reminder of everything I don't know — a key driver for continuous learning.

#9: I don’t know how to resolve the tension between the desire to enjoy the world, the curiosity to understand it and the rage to change it

I wrote this note in my journal on the morning of 16, February 2021, recalling the events of the previous day. A young environmental activist was arrested, and it triggered a series of thoughts laying bare the inner conflicts I often deal with. I don’t have any meaningful way to resolve them, but confronting them is perhaps a good starting point. This is a slightly edited version of the original note in my diary. 

Yesterday was weird.

I started the day with a run; went to work; posted an Instagram story about the cheap thrill of getting a free Blue Tokai coffee (thank you, WeWork!); wondered if my caffeine intake is beyond limits, and felt greatly relieved after learning that 400mg of caffeine per day is an acceptable amount—meaning four cups of coffee are not going to kill me; told my colleague we should order a croissant in the evening from the bakery he had introduced me to; wondered if I should unwind for a few days in Goa before heading back to home in Delhi; messaged a few journalist friends regarding an idea of a course I am thinking of teaching. Usual everyday stuff.

Cut to the evening. I was outraged: the Delhi police had arrested Disha Ravi, a 22-year old environmental activist for creating a toolkit intended to wage "economic, social, cultural and regional war against India"—just another bogus case to silence dissent. 

The absurd executive actions of the Indian state and its arbitrary exercise of power do not surprise me: it’s a feature, not a bug. But the form in which it manifests itself touches new lows every time. "How can they do this? This is insane!" are the words I routinely scream inside my head. 

Just days before I lost my cool after reading a story in the Washington Post that reported the findings of a new forensic report telling us how a "key evidence against a group of Indian activists accused of plotting to overthrow the [Indian] government" was maliciously planted by malware on the laptops seized by police. Again, it’s not that I am not aware about the hunger of the Indian state to spy and surveil its citizens—I write about that stuff—but the specific incidents reveal what's at stake. 

I don’t see each of these incidents in isolation. In aggregate, it feels like India is an illiberal democracy with a liberal constitution gradually backsliding to an authoritarian state. 

But I also have doubts about this assertion: what if my analysis is wrong? I have a really poor understanding of history—I am trying to fill that gap—and that makes me unsure if what I am seeing right now is a downfall or if the Indian state has been like this forever. Whether the incidents that trouble me in Modi’s India are directly related to this regime or if they are a constant feature of India's governing class? I wonder if all my ideas about the India I had imagined as a kid were actually wrong, and if the information I had been fed was riddled with propaganda? I worry if the majority of Indians don't align with fundamental liberal values that form the cornerstone of Indian constitution. And what if all of this is not about India at all: maybe the layers run deeper and hint to the relationship between those who exercise power and its subjects?

I really don’t know. I spend a lot of my time scratching my head and turning to books looking for answers. This is the part of me that is really curious to understand the world and build a model to make sense of it. 

When I think about these questions, I try to avoid daily news and social media chatter. They make it hard to isolate events from systemic processes. But I also realise that not looking at these discomforting incidents does not mean they will go away. This is the world we live in and these incidents are happening right now. The frustrating thing is the sense of not being able to do anything about it and witnessing the world as a mute spectator. 

Which raises more questions: What should I do? What can I do? How can I uses my skills and abilities to make a positive contribution? What is the philosophical frame in which I should think about my role as a citizen and separate it from that of a free individual?

I don't think I am going to get any meaningful answers to these questions anytime soon. In the interim, I want to figure out things that would satisfy my conscience, like donating a fixed proportion of my annual income to charity. It also includes exercising my inalienable right to raise voice against excesses of the Indian state every time I feel the urge to do so (I am carefully using these words). Especially so in the offline world, meaning participating in protests to show solidarity with causes I care about. 

So yesterday, at 3.45pm, when I saw on Twitter that people are gathering at Mysore Bank Circle calling for an immediate release of Disha Ravi, I instantly decided to go. 

The scheduled time was 4pm. The croissant we ordered had just arrived. I quickly ate it, booked an Uber and rushed to the protest spot. I reached around 4.40pm. At first, I was elated: there were real people out there, protesting against the illegitimate arrest of the young activist. I clicked photos, shot videos and tweeted a few. But just when I thought more people would arrive, the opposite happened: around 5.30pm, the crowd dispersed. It was over. Huh?

“This city is so disappointing,” I texted two friends. “Just 100 people show up? Really?”

A fellow protester said that 100 people turning up for a protest in Bengaluru on a Monday evening was actually quite good. It is possible that many were not aware of the gathering, or a short notice meant they could not leave the workplace, or they were generally avoiding crowded places due to the pandemic. Made sense. But my disappointment stayed. 

I booked an Uber to get back. On the way, I played The Trial of the Chicago 7—a movie that had a deep and profound impact on my thinking. People around me seem to have lost a sense of civics—of being a citizen—and the movie was a powerful reminder of what that means. 

Ten minutes later, a friend who has gone abroad to study public policy texted me and said he wanted to chat. He was struggling with similar questions. We got on a Zoom call in an hour and discussed the state of the world and where we see ourselves positioned in the larger scheme of things.

We discussed what we always do: about the deep privilege both of us hold that ensures there is zero impact on our daily lives with actions of the Indian state; the discomfort we feel for being in that position and not knowing how to act on those feelings. Both of us realise that we must do something but we believe we should not rush: the quest to make the world better is not a sprint. We should learn, listen to people, go to the ground, fill gaps in our shallow understanding, eliminate myths—all that. While we feel unsure what that "do something" should be, the tension is what if it gets too late? 

We spoke about our common friends who treat political discussions as nothing more than evening gossip. “Yeah man, it’s terrible” is all they say—it doesn’t trigger any thoughts or change in actions. We reminded ourselves that we are asking these questions because we care. These incidents trigger us: we feel more determined to level up our efforts every time the government does something non-sensical. We must sensibly process this outrage.

All the forces around us would tell us that we are over-thinkers, idealistic, misguided privileged liberals ranting but doing nothing, that we should stop bothering because nothing can really change. We have to actively resist those voices. Deep inside our hearts, we know it would be hard for us to live a fulfilled life if we give up on our deepest convictions about the possibilities of a better world — for everyone. Opting this path is not easy and we knew that when we chose careers that overlap with civic life. 

But giving up hope for change means surrendering to cynicism—which is quite different from being skeptical, an acquired trait of good journalists—and that's not a version of myself I would like to see anytime in the future. 

The call ended. I packed my bag and started walking towards a beautiful cafe I had decided to spend my evening at. 

The events of the day really confused me: how do I go from getting excited about free coffee in the morning to being concerned about the state of the country in a matter of a few hours? What explains this duality? What gives me this superpower to switch between these two widely different modes of thinking about my place in the world? 

It’s like the difference between my Twitter and Instagram feed. On Instagram—which I started using only in the last few months—I follow five types of accounts: friends, celebrities, bookstagrammers, dancers and fitness instructors. The Instagram version of me wants to enjoy life: dive deep into a great book, spend quality time with family and friends, travel to the mountains, explore new things, pursue my hobbies, and do all the small things that make life worth living. Then there is the Twitter version of myself which gets angry looking at the unjust, unfair and unequal world. A fire rages inside me to do something to change the status quo.

How can I move between these modes as if I am just alternating between browser tabs? I struggle to make sense of this. At the core of it is The Big Question: How do I balance my desire to enjoy the world, the curiosity to understand it and the rage to change it?

I was thinking all this while sitting at the corner table of that lovely European-style cafe with an outdoor seating. Then and there I decided to cancel the Goa trip I was contemplating in the morning. I could not do it. Not that I would do anything radically meaningful by skipping a 3-day holiday—it would make zero difference to myself or anyone else. It’s just that I did not have the courage to make those bookings and unwind at the beach while I struggled with these internal contradictions.

I walked back home, took a shower, laid down on my bed, still conflicted, wondering if and how the thoughts of the day should trigger any action or changes in what I do on a daily basis. Unable to process it, I turned off the music playing in my headphones and jumped to a 17-minute audio clip I had recorded for myself in October 2020: a monologue of me talking to myself, reminding myself of my privilege, and the various high-tension points in my life where I made some hard calls, where I followed the path my principles directed me towards, recounting how that made me feel, why I should not leave that path and stick to my values. Those reminders are important. 

My mind was probably exhausted by then: I fell asleep within minutes, and most of that dialogue remained unheard. I realised this morning how comfortably I had slept—despite all those anxious questions occupying my mind! How? I don't know. It was just another contradiction, a perfect illustration of what the previous day was all about.



#8: How I cope when it seems like the world is falling apart

Totally lost my cool yesterday evening. The trigger: a story published in the Washington Post. I posted an impulsive update on Instagram.  

This morning: I could not sleep well. One of those days when I woke up feeling a bit anxious, thinking about all the easy, comfortable and privileged choices I have made in a deeply unequal, unfair and unjust world. 

So I got off the bed, went for a quick run, hit the gym and lifted weights. The hour-long workout, as always, helped me get rid of anxiety and prepared me for the workday. 

Then I reminded myself: the struggle for living in a better world is a long, arduous process. Don't react: understand and act. The only thing I can do in the short run is to give my best in what I do on a day to day basis. Resist things in smaller everyday battles. Fight against unjust power structures lurking in family, community and workplace. Don't accept the status quo. Support fellow citizens actively fighting for causes I care about. Keep eyes on the bigger battle. That's in my control. 95% of the things in the news are not. 

If that doesn't work, go back to the drawing board and reconsider the choices I made, and change course. What else?

Most importantly: whatever be the case, stay hopeful—even if that requires some elements of self-delusion :)

#7: Turning politics into theatre

Tweets by an international pop star—five words, one hashtag, a linked article—and a teenage climate activist on India's farmer protests have rattled thin-skinned Indians. What an irony: A citizenry perennially subjected to state-sponsored propaganda is tweeting #IndiaAgainstPropaganda. 

Liberal critics, unsurprisingly, are outraged at this mass reaction and calling out, among other things, Indian celebrities for "towing the line", asking them to "show some spine" instead of participating in this non-sensical tamasha. 

This incident reminded me of a fascinating interview of documentary film maker Adam Curtis published in The Economist magazine in 2018:

No one is really sure what Trump represents. My working theory is that he’s part of the pantomime-isation of politics. Every morning Donald Trump wakes up in the White House, he tweets something absolutely outrageous which he knows the liberals will get upset by, the liberals read his tweets and go “This is terrible, this is outrageous,” and then tell each other via social media how terrible it all is. It becomes a feedback loop in which they are locked together. In my mind, it’s like they’re together in a theatre watching a pantomime villain. The pantomime villain comes forward into the light, looks at them and says something terrible, and they go “Boo!!”. Meanwhile, outside the theatre, real power is carrying on but no one is really analysing it.

This is the problem with a lot of journalism, especially liberal journalism at the moment. It’s locked together with those people in the theatre. If you look at the New York Times, for example, it’s continually about that feedback loop between what Trump has said and the reaction of liberal elements in the society. It’s led to a great narrowing of journalism. So in a way, he is part of the hypernormal situation because it’s a politics of pantomime locked together with its critics.

And it becomes a perpetual, infernal motion system, which is a distraction. It’s not a conspiracy. It’s a distraction from what’s really happening in the world. I would argue that there is a sense—in a lot of liberal journalism—of unreality. They’re locked into describing the pantomime politics and they’re not looking to what Mr Michael Pence is really up to, and what’s really happening outside the theatre.

Mr Curtis is on point. This is a circus looping on auto-pilot with a predictable script. The story may begin with a different incident, new actors may appear but the outcome is always the same: nothing happens, nothing changes. The reality lies elsewhere—ignored. 

#6: Journalists refer to themselves as storytellers. Is that a mistake?

"Facts come in the way of beautiful prose," a colleague at the Wall Street Journal told me during my 2018 fellowship at the newspaper. 

It stayed with me. Every time I ponder over the limitations of journalistic methods to understand the world, I think about it. Especially in the last two years as I switched gears to work on more ambitious stories (relative to what I was doing earlier) with no easy answers. Even after tenaciously chasing all possible sources of information (data, documents, interviews, digital trails etc), piecing together a definitive story—say about a specific incident, a policy decision, an organisation—is hard. Very hard. 

How do you know for sure? 

In most cases, I operate on Bayesian probabilities. For an open-ended question, I start with a prior—my hypothesis on what I think is happening. Then I report. The new information moves the needle. It strengthens or weakens my prior. I try finding counter evidence for my hypothesis to stress test it. As reporting continues, some pieces start falling into place. The degree of belief for a few specific things starts increasing: most evidence points in the same direction and alternative explanations are proven wrong. 

Yet, the uncertainty remains. What have I missed? What is it that I am not looking at? These conflicts will remain and it is okay. It is an integral component of any knowledge production process. 

But things get tricky when you sit down to write. The most readable and enjoyable stories follow a clear narrative arc. Take fiction writing. The world being described exists inside the mind of the writer. They know everything: they are only bounded by their imagination. 

True stories are different. Journalism is different. Which is why I sometimes feel conflicted about the promise of narrative non-fiction genre. Won't facts and uncertainty play spoilsport when one is writing novelistic narratives with cinematic scenes? How do you handle that? 

This remains an open question for me and I was thinking about it last weekend. I wrote to my former WSJ colleague (referred above) to get his thoughts. His response was insightful and thought it's worth sharing. Here is our email exchange:

1. From me:

I have thought a lot about this over the years and it is a bit worrying. Almost all deeply reported stories have that defining paragraph (nutgraf) to signal why this story matters and why should readers spend their time on it. But many times, I see writers making those "larger points" that sound perfect in theory, but they may be stretching a bit beyond what the reporting material is revealing. There remains a subtle threat of overselling.

So I constantly remind myself that it is okay to not have that amazing paragraph (as much as I would love having it) where all words neatly add up to delight my readers. The nuances, the ifs and buts, matter. However, so many journalists feel they know-it-all and write with such great authority that it automatically lends itself to beautifully written prose.

Is this conflict widely acknowledged in the American press? I would love to read more on what editors think about it. 

2. From my former WSJ colleague:

I'm not sure that there has been any great introspection on the part of American journalists when it comes to these issues. I think Tom Wolfe, Truman Capote, and Gay Talese helped lead journalists astray with their riveting, but dubiously sourced, "New Journalism." It was exciting and eminently readable, but couldn't possibly be totally true and correct.

I often hear journalists refer to themselves as storytellers, but I think that is a huge mistake and leads to the kind of overreach and crafting that you mention. We should see ourselves first as fact relayers, and we should be open and honest about what we do and don't know and why our "story" doesn't follow a perfect arc.


We should clearly source everything. We should clearly and generously credit those whose work informed ours.We should always understand that the "why" part of the five w's is the one that we can never be close to sure about. How do we presume to get inside an actor's head and understand his motivations, or that we can account for hundreds or thousands of random events and people that come together at a moment in time and lead to something happening?

I always have these thoughts when I read foreign correspondents who have parachuted into a foreign land and confidently assert the cause-and-effect of endlessly complicated situations. I would never have the self assuredness to do that.

I love the term sophomoric, because we all go through that phase of learning so much that we think we know it all. One nice thing about growing older is that you really have learned a lot, but you also have learned just how much you don't and might never know (if you're paying attention at all).


#5: How I approach and manage my freelance journalism career

"There is a lot of uncertainty in figuring out how this will work out," reads the seventh point in my journal entry titled "On Quitting", dated 13th January 2019—the day I resigned from my full-time job at the Hindustan Times

After two excellent years at HT—where I got the chance to learn from and collaborate with a bunch of talented and hardworking journalists, where editors gave me the opportunity to hone my craft, space to write meaningful stories and nominated me for an amazing fellowship at the Wall Street Journal—I was done with it.

As in most jobs, my stint at HT had both upsides and downsides. I struck a balance, making most out of the opportunities at hand, downplaying stuff that troubled me, and moving on. But everything has a threshold. Somewhere around December 2018, I knew it was time to leave. I was not fitting in. If I wanted to stay inspired to be a journalist, I had to quit. And so I did. 

At the time of resignation, I had no plan for what comes next. "How long? Who will read? Who will pay? What next? Am I in the right profession?" I wrote in my journal. "These questions come naturally when you know that salary won't be automatically credited to your account," I added, anticipating my cash flow in the coming months. "It does not sound ideal. But at least it is forcing me to think about my priorities. And that’s good."

Two years later, with the benefit of hindsight, I am so glad I took that call to quit my job. I did not go back to a full-time role at a news organisation—by choice, not due to lack of opportunities. I now work as a freelance journalist: In 2019, I exclusively did reporting and writing; in 2020, I started with a dual career, splitting time between freelance journalism and software engineering. 

I love working independently. I earn enough to pay my bills, save and invest; I am getting closer to the kind of work I aspire to do; I get a hell lot of time to learn new things, read, relax, pause, sit and just think; most importantly, I can prioritise stuff I had long ignored in early adulthood: my physical and mental health.

In the last two years as a freelance reporter, I have written long stories for various Indian and International publications. You can read my published work here

I am writing this post to share my experience. 


1. Background: We are not playing the same game

I hesitate to offer freelancing advice. Consider this: When I quit my job, I was 25, had a year's worth of emergency fund, no debt, no EMIs, and because I live with my parents, no rent to pay. That means, even if things would have gone downhill, and nothing worked out, I could still comfortably get through another year. This cushion allowed me to wait and see how freelancing goes, rather than immediately look for a new full-time job. 

Add to that the two intangible assets I own: an elite undergraduate degree (mathematics and scientific computing from IIT Kanpur) and high-in-demand technical skills (building software, data science). I think of them as insurance instruments to protect me from any future shocks. In the winner-take-all economy—which favours the elite and perpetuates inequality—my educational background and interests have equipped me to enter that elite group. (There is a great deal of discomforting dissonance between who I am and the vision of the society I want to live in. But it is what it is.)

So simply put, I have a lot of structural privilege. I am fortunate to not have to live with survival questions. If all hell breaks loose, I always have the option to build a great career in the technology industry. I can't stress enough how much this matters in every single thing I do. 

I mentioned this to make one thing clear: my worst-case scenario is very safe. As much as leaving a job without a plan sounds like a risk, it wasn't a big deal: the stakes were low. 

It is not so for most people.

Moreover, multiple factors will dictate your experience: social capital and network (people you know), skills (what you know, what you can do, how you fare with competition), random chance events (on which you have no control), your expectations from life and how you perceive risk.

It is important to keep this context in mind as you read further about my experience. I have figured out a model that works for me — at least for now. But no two people can have an identical model: we are not playing the same game. There is no right approach. It varies with who you are and what you want. What worked for me may only be partially applicable to your scenario, or perhaps not at all. 

This post only describes my experience as an individual. 


2. Framework: What do you want?

Goal clarity is the single most important thing to deal with the inevitable uncertainty of a freelance career. The future is unknowable, and plans rarely work — beyond six months, no rigid plan has ever worked for me. So my approach is to operate within self-imposed rules that align with my larger abstract goals. They guide my decisions and help me choose. Especially so in thinking about trade-offs, and in rejecting lucrative opportunities with a high short-term appeal. 

Most of my goal-setting process revolves around having clarity on five sets of questions:

1. Why am I a journalist? Why not any other profession? What do I expect from my work?

2. What kind of journalism I want to do? What is that I do not want to do?

3. Why am I freelancing? Why not find a full-time job at a publication? What can I do in a big media setup that I can’t do independently, and vice versa? What are the trade-offs involved?

4. What are my needs and wants, right now and a little ahead in the future? How much money do I need to fulfil those? What is my personal finance philosophy?

5. What kind of life do I want outside work? What are the things I value in my life? What are the things that matter to me? What are my core underlying principles? 

These are hard questions. I struggle answering them. They force me to ask deeper questions about my belief systems, most of them hidden from immediate view, wrapped in dominant ideologies of the day. But over several months, scribbling in my journal, I figured where I stand at the moment, helping me set clear personal and professional goals. Of course, these answers are not set in stone and will change with newer life experiences. 

Here is a brief glimpse: I don't find it thrilling to get a lot of bylines. I enjoy writing long stories, exploring one topic in depth. I can't recklessly follow the news cycle. I prefer working with good editors who help me improve my craft than getting published in a big publication for the sake of it. 

I don't care about the "impact" of my work—I have no control over that. But I do have control over effort and integrity, and it matters to me. Reporting rigour in stories should follow an upward trajectory. I should learn new things. I value journalistic freedom more than anything else, even if that comes at the expense of readership. 

I am incapable of maintaining an active social media presence, especially Twitter. High-quality leisure is important, and I don't want my work to take over everything else I care about. Ethics and principles remain at the centre of my decision making, and I try to reduce the inescapable moral contradictions we all live with to whatever degree possible. 

    The list goes on. 

    Freelancing allows me to curate my career to fit many of these preferences and gives me more control over my time. That's the reason I stuck with it. 

    I did apply for one full-time job in the last two years. A couple of months after I left HT, the Wall Street Journal had an opening for a cyber-focused investigative reporter and an editor at the paper—who mentored me during my 2018 fellowship at the Journal—suggested I should check it out. I applied, the process began, but it didn't work out. I didn't apply for any full-time position after that. 

    I do have a few dream jobs and at some point in my career, I may try getting there. Not now, though. It doesn't feature in my six-month goals—the maximum time I plan for. 


    3. System: How to achieve your goals?

    Goals are crucial. Equally important is to have a pragmatic system to take you there. This includes the specific mechanics of getting assignments, managing money, balancing multiple projects and meeting deadlines. 

    Here is how I get reporting gigs:

    1. I write infrequently and mostly long-form reported stories. There are many publications interested in original reporting, and I am lucky that story ideas that I find exciting overlap with the interests of many publications which have budgets to hire freelancers. 

    2. Sometimes, an editor approaches me with an idea, or just an offer to contribute to their publication. In other cases, I pitch ideas to specific publications, checking if they would be interested. It goes both ways. 

    3. I started freelancing after working at India's largest publicationsThe Hindu and Hindustan Times—for three years. So I knew a couple of editors who hire freelance writers which made it easy for me to get started. 

    4. What has worked the best: building solid relationships with a few editors. Keeping in touch about ideas and writing for them once in a while. These editors have a good understanding of how I work and what they can expect if they commission an assignment to me. I strongly believe that the key variable predicting a journalist's output over the long run is their reporting process, everything from ideation to writing. I have been working hard on improving my process—a work in progress—to tackle complicated stories. Editors who know me see that in action and that sets the right expectations for a mutually beneficial arrangement. 

    5. I like to think of myself as a beat-agnostic reporter: not restricting my reporting to a few themes and chasing stories that pique my curiosity. That's ideal. But it's good to have a niche that defines why an editor should reach out to you. In my case, you will see three strands: reporting at the intersection of technology, politics, policy and culture; long-term investigations; and data journalism. This combination has limited supply in a crowded journalism market and helps me in getting assignments. 

    6. I always ask myself: If I get to tell a story, what is the value I am bringing in? What will make my story stand out if five different reporters are simultaneously working on it? How can I be authentic? If editors can see that value, the chances of getting another assignment increase. 

    In the last two years, I have not seen a phase where I had to struggle to find work. I hope it stays that way. 

    On balancing projects/timelines: I am not good at this. Things keep piling up, the todo list keeps extending, leading to weeks where I have to continuously slog to wrap things up. I am working on this and learning how to manage my time better. 

    Financial management deserves a detailed discussion. 


    4. Money: How do you sustain?

    "Huh, so you are living off your parents' money," said a senior journalist, whose work I admire, when I mentioned I work as a freelancer. I would not be surprised if many others believe the same. I just wish they also factor in that journalism is not the profession most Indian parents would willingly cross-subsidise. Not mine, especially when their kid has a STEM degree. The day I graduated from college, I knew one thing: the only way to resist parental pressure is to remain financially independent. 

    But I understand why they think so. The most common question friends ask me is about money. "Do you get paid enough?", "How do you sustain?", "Isn't it risky?"

    Let me turn the table for a moment: Do you feel secure at a large media organisation? What about layoffs? How do you feel when your annual raise is less than the rate of inflation? 

    The point is simple: you can't escape risks. But you can find ways to manage them.  

    Look, I don't want to make a killing fortune — and if you do, please don't become a journalist. Wrong profession for that goal. But neither do I want to be miserable at any point or worry about my finances all the time. I don't have too many material desires, but I am no monk either. What I need is a way to strike that fine balance. 

    1. The question that drives my thinking around finances: how I can do what I want to do while earning enough? Being a freelancer is like running a one-person business. I do not care about the larger economic crisis in journalism on a day to day basis: all I want is a publication willing to pay me, fairly, for my next assignment.

    2. My strategy: I start by asking how much I need. I assign myself a modest annual package—Rs X lakhs per year, just like a salary at a regular job. With the amount set, I work backwards: what do I need to achieve that X? Then I fill the bucket as much as possible with earnings from the stuff I like to do. For the remaining sum, I don't care as much about the specifics of the work: I just need to fill the bucket to make my system work. 

    3. Finances in 2019-20: This approach worked out pretty well for me. In the first year, I made almost all my money exclusively through writing assignments. That's what encouraged me to continue working as a freelancer. To give you a sense, in the first year of freelancing, I set myself an annual target equal to 90% of what I was making at the Hindustan Times. 90% because I did not want to get too ambitious about finances right in the first year, as I was figuring out how all of this works. But neither did I want to keep it too low and move forward with an unrealistic evaluation.

    4. Payment structures: Many times I have been paid similar per-word rates for a story I wrote in one day and another that took me three weeks. It makes no sense, but many publications only operate on word-count based rates and you can't do much about it.

    Some publications pay you for your time. For instance, in 2020, I signed up for a two-month exclusive arrangement with HuffPost India where I would be paid a monthly fee, irrespective of the number of stories or their length. I had a similar contract with the India Today group, also for a couple of months, to cover the 2019 general election. 

    5. Cross-subsidise one assignment with another: Foreign publications pay significantly more than Indian publications—even though the effort might be similar. A higher payment from one assignment enables me to spend more time on relatively lower-paid assignments. 

    6. Rates: the per-word rates in Indian publications average around Rs 10-15 (good ones – some pay even less than Rs 5 per word or nothing at all). American publications pay in dollars, and I have seen rates ranging from 0.5 to 1.5 USD. For example:

    PublicationRates
    HuffPost India 
    (not operational any more)
    Rs 10 per word/custom rates for special assignments
    Mint Long StoryPre-pandemic: Rs 30,000 (~2000-2200 words)

    Now: Rs 20,000 (for same stuff)

    The Atlantic$0.5 per word (~Rs 37 per word)
    Rest of World$1.5 per word (~Rs 100 per word)


    7.
    Delayed payments: A quick Twitter scan shows me delayed payments are a norm in the industry. Fortunately, I have almost always been paid in time. My first payment from Mint was delayed by a few months — but that was the last time it happened. 

    8. My side job:
     Many freelancers I know have some sort of a side job, leveraging their core skills elsewhere. For instance, a journalist friend splits her time between reporting/writing (core work) and working for a big data startup (side job) who hired her to write stories based on insights from their analytics for their internal use. A freelance musician I met at a cafe splits his time between producing music (core work) and teaching music at a university (side job). And so on.

    The idea is to diversify your income source and not be dependent on writing alone. I got on a side job mode in my second year of freelancing, leveraging my programming skills. 

    In January 2020, a friend and I started PopperAI, a news automation company. Our software created thousands of machine-generated articles based on high-frequency public data. But we shut shop in July due to various reasons. The purpose of starting the company was clear: it was a means to an end. I wanted to build a parallel high-value revenue stream for myself which would buy me more time for reporting ambitious stories. Alas, it didn't work out and my financial model for 2020 took a hit, albeit temporarily. 

    I then explored other freelance opportunities in software engineering and got a six-month contract with a Bengaluru-based startup, where I now contribute part-time. It's a great deal, especially because the company gives me the flexibility to work on my schedule. 

    I find the side job very empowering: it drastically reduces uncertainty and helps me focus. The fact that I enjoy writing code and thinking about engineering problems makes it the best of both worlds. 

    9. Building a healthy and thoughtful relationship with money: carefully thinking through my personal finance philosophy helped me a lot. Maintaining a low burn rate and a high savings rate, creating an emergency fund, planning for short-term/long-term investments, buying a good health insurance policy—all the standard things people recommend to make money work for you. Consider reading Morgan Housel's book Psychology of Money and Monika Halan's Let's talk money to educate yourself. It's so important. It took me some time to get in that flow. 


    5. Other things: what is it like to be a freelancer?

    - Where I work: I have been working from home long before the pandemic made it a compulsion. But I am more productive when not at home. In normal times, I work from coffee shops. There are several work-friendly cafes in New Delhi and I identified a few spots where I am a regular — who let me sit through the day even if all I order is one cup of coffee. I considered moving to a co-working space at one point. A flexible seat costs around Rs 8000 per month. But I decided against it and stuck with cafes. 

    - Become a digital nomad: If you are a nature person, go work from hills. It's amazing. Places like Zostel Mussoorie—one of my favourite spots—are designed for remote working. This will only grow post-pandemic as both the supply and demand are likely to increase. 

    - You have to hold yourself accountable: In a regular job, you are told what your mandate is. You have an editor who has expectations. You have a fixed salary. Freelance changes that. You have to decide everything for yourself. You get to decide what you want to do and how. You can direct your career to the style of journalism you prefer. You decide the extent to which you will report a story. You decide the bounds. The freedom can be daunting if not leveraged thoughtfully. It's not easy.

    Work is no longer the central part of my life: The institutions we are part of and the work we do there become a part of our identity. That erodes in many ways as a freelancer. There is no concept of weekdays or weekends. Every day is different. For instance, if I am desperate to finish a new book, I can just not work for two days and read. Or meet friends in the afternoon and work late in the evening. Then there are weeks where I am packed with assignments and have to work every day. I can plan a two-week trip to the mountains and then take more work when I come back. I love this flexibility.

    - You only have control over your own story: When employed at a news organisation, I think beyond my work. I care about the newspaper I am part of and try contributing, in small ways, to improve the overall coverage of the publication. On big event days, I loved collaborating with the team. I could raise red flags about the work we do with folks who make editorial calls. In short, you can contribute to building stronger institutions. Freelancers don't get to do that. You operate as a solo journalist and all you can do is do a good job on your article. 


    I hope this was helpful for you. Questions or comments? I am at samarthbansal@protonmail.com. For occasional updates about my work, sign up here: bansalsamarth.substack.com


      #4: How interlinked economic and political forces create self-censorship in Indian media

      Most contemporary discussions on press freedom begin with some sort of rankings: X country slipped Y positions on Z index—that’s evidence something wrong is happening.

      This makes headlines every year in India, as we continue to slip down in these indices. Criminal defamation cases are filed against journalists for merely doing their jobs and reporting stories. Organized disinformation campaigns and troll armies delegitimize the institution of journalism and individual reporters to evade trust in the media. The list goes on. These observable metrics—all problematic—reflect in the various rankings.

      But I don’t find much meaning in this data. As someone who plays with numbers for a living—I am a data journalist—I remain deeply skeptical about any methodology that claims to quantify press freedom. It’s hard. That’s because the everyday experience of being inside a newsroom, the anxieties journalists live with and forces that shape editorial decisions stay hidden from the outside world — all they get to see is the output, which, however egregious, doesn’t capture the true extent of the rot in the Indian press.

      Let me give you a brief glimpse of a few revealing aspects of the big media. I will restrict myself to English language publications with the largest readership and circulation.

      Of India’s four largest English newspapers — Times of India, The Hindu, Hindustan Times and Indian Express — only Express has a dedicated investigative reporting team, which, for instance partners in international stories like Panama or Paradise papers. Their team strength: less than five.

      None of the other three papers even have the position of an “investigative editor”. That does not mean the papers don’t investigate or publish critical stories — or, as the public opinion goes, “sold out” to the government. It’s not that simple. Their coverage is crucial to our understanding of modern India and reporters at those papers are doing important work. But the point is, hard hitting stories are rare, and when they appear, it’s driven by the efforts of a few individuals pushing boundaries rather than a systemic culture encouraging publication of uncomfortable stories. Don’t judge a paper by what it publishes. What is excluded is generally more telling.

      In September 2018, I returned to India after spending five months reporting with the investigative reporting team of the Wall Street Journal in New York as part of a fellowship program. I was pumped up to use my newly acquired skills for reporting in India. However, on the very first day, when I started meeting journalist friends, I saw a drastic cultural difference: at the Journal, you pitched an investigative idea, and colleagues helped me brainstorm how to report the story; in India, colleagues said no point chasing this story as it won’t be published.

      That’s not without reason. Consider this: I reported a story showing how propaganda is disseminated and troll accounts are promoted on the Twitter-like interface in the official app of Narendra Modi, India’s Prime Minister. My editor sat on the story for two months. Why? He was busy. But he did have time to edit the other stories I was filing. No explanation was offered.

      In another case, I wrote a story that showed how the Twitter metrics — retweets per tweet, for instance — of India’s opposition leader Rahul Gandhi had surpassed that of Mr Modi. All hell broke loose for publication of such a benign story (it took me half a day to analyse and write this short story). Union Ministers from Mr Modi’s party accused my report of propagating partisan propaganda. Next day, I got a phone call from the owner of the publication—whom I had never met—asking me to follow up and check if the opposition leader’s metrics were amplified by a bot. I should send the story directly to the owner and not to the editor. Sigh. It is not too hard to imagine what would have transpired behind the scenes: actions that remain a mystery for reporters at my pay grade.

      This toxic culture of self-censorship has become oblivious: many journalists may not even recognize the boundaries they have constructed in imagining what a story could be. Once, I was given an assignment to analyse the content of the Prime Minister’s campaign speeches. The editor wanted numbers: how much time, as a proportion of the overall speech, does Mr Modi take shots at the opposition than talking about his promises? It was a fairly analytical story. I had to randomly select a few speeches, make sure I cover different time spans and states, listen to the speeches, create a dataset, and tell what I found. A senior political reporter was asked to help me. The first question the reporter asked: “have you been told what the analysis needs to say?” I was shocked. The implications were clear: it was normal for that reporter — at least ten years older than me — to be sometimes told what the story needs to say before reporting it. That’s not acceptable.

      It took me some while to understand how common this is: some reporters start doubting themselves and ask whether they are chasing stories that don’t matter, whether they have not done the job properly, or simply the institution they work for is compromised. The answers don’t come easily. Others, who don’t think too much about these supposedly “idealistic” questions, continue to flourish. “Not a single story I wrote was ever censored,” they will tell you. Of course—it won’t be.

      This form of self-censorship, I would argue, is a bigger problem than explicit censorship. Because it happens subtly and parasitically cripples the institution of journalism. It’s driven by interlinked economic and political forces. For instance, most of India’s large media houses that are struggling with cash organize grand annual conclaves to bump up their bottom lines: Political leaders, top company executives, sport stars, and Bollywood celebrities are invited as guests. The bigger the names that speak at your conclave, it is said, more money sponsors put in.

      But what if Mr Modi is in your speaker list, and he refuses to turn up at your event at the last moment? It could be disastrous for the company’s finances. This is not a theoretical case: it has happened with one of India’s largest papers. “Security concerns” was the official reason, but news reports flagged the PM was not happy with the paper’s coverage. Which simply means you can’t offend Mr Modi’s sensibilities for at least a month preceding the event. Imagine the pressures a reporter would face to even pitch a story that holds the PM accountable.

      I have deliberately chosen all the examples that relate to Mr Modi. In journalism circles, it is often said: you can criticize the government, write about his party, but things get complicated when the story centers around the Prime Minister himself.

      George Orwell wrote: “Intellectual honesty is a crime in any totalitarian country; but even in England it is not exactly profitable to speak and write the truth”. With all signs of India sliding towards an illiberal democracy, the situation here is no different.

      So, hypothetically speaking, even if India’s position in press freedom rankings jumps a few places, it would be no cause to celebrate. Unless the systemic problems which one finds inside India’s largest newsrooms begin to change, and reporters and editors can think and write freely, while being committed to the craft and core values of the profession, free press in India will remain a myth.

      Originally published at n3Con magazine

      #3: Thoughts and observations on data journalism in India

      Today, I complete five years in journalism: one year at The Hindu (my first job, straight out of university), two years at the Hindustan Times—which overlapped with a five-month fellowship at the Wall Street Journal—and two years as a freelancer.

      "Data journalist" was my official job description in full-time roles. But I find that term redundant and I don't like to use it anymore: all journalism should be data-informed and evidence-based. I prefer calling myself just a "reporter". In part, this identity shift—primarily in my head, I doubt if anyone else cares—is a reflection of the changing nature of my work: I have moved from writing newsy analytical stories to a mix of investigative and narrative reportage. Data and empirical thinking continue to be at the centre of my work, but it's no longer defined by it.

      This post is about data journalism in the way most people understand it: journalists who specialise in producing data-driven stories, a task that includes everything from collecting data to analysing and presenting it visually.

      Here are some thoughts and observations on data work in Indian newsrooms.

      1. Data journalism is an umbrella term which means different things to different people: adding data-backed context in daily news coverage; analysis of issues dominating the news cycle; making infographics with interesting data factoids; narrating stories and explaining complex concepts using visual graphics; using hard data for investigations, highlighting previously unreported trends and throwing light on specific incidents; empirical coverage of big-ticket events like elections and budgets.

      I have done all of this. It's useful to think of them as different tasks as the skillsets vary widely. A lot of stuff can be done without writing a single line of code or making pivot tables in spreadsheets. But the fanciest stuff we do gets the most attention, making regular data work appear more complicated than it is.

      2. Data journalism is not just about data visualisation or interactive stories: I categorically mention this as a separate point because I used to conflate the two when I started. Back in 2015, data journalism for me was the visual stuff the New York Times published. I would browse their interactive stories, think about how we can take inspiration for telling similar stories in India, rant about the technical restrictions our archaic online publishing systems posed, and ultimately make peace with the baby steps we took in creating an ecosystem that valued interactive storytelling.

      I learnt my lesson soon: I was getting too attached to the form. Yes, charts are nice. Interactives, when thoughtfully designed, can help tell brilliant stories. But that should not divert our focus from the fundamentals, which is the data itself and the meaning it's adding to the story. Once you get past that stage, you can figure out how to stitch words and visuals to communicate your findings to the world. Sometimes, one small static chart can tell a story that no words can capture, and in other cases, data leads you to nuanced explanations where clear prose becomes crucial.

      3. Visual journalism in India can wait: If a newsroom has infinite resources, sure, they can—should—hire a team of a dozen or two coder-journalists with excellent programming and design skills. Let them work on exciting special projects, fully leveraging the interactive nature of the internet to tell relevant and compelling stories. From mid-2016 to mid-2018, our small data and visuals team at the Hindustan Times—synonymous in journalism circles with bylines of Gurman Bhatia and Harry Stevens—did just that. I loved the work we did, and I learnt a lot. Bhatia has since moved to work for Reuters in Singapore and Stevens is at the Washington Post.

      But over these years, as I have been thinking more broadly about Indian journalism, I have become increasingly convinced that we are not yet ready to institutionalise this kind of work. Cash-constrained Indian newsrooms neither have the resources to hire that talent nor the imagination to nurture it. Yes, things have to start somewhere, but I don't see that time is now.

      Instead, we should be prioritising work that produces original reporting and boosts collective news-gathering efforts. Newsrooms should hire journalists who can code and ask them to think creatively and build original datasets, ideate on innovative ways to use statistics in identifying ignored trends, invest in long investigative projects. While the focus remains on sharp data reporting, newsroom nerds can occasionally produce visually appealing journalism.

      Journalists should still make simple, effective and aesthetically pleasing charts using simple tools like DataWrapper, which have evolved significantly in terms of capabilities over the last few years. But for now, a dedicated focus on building a culture of interactive and visual journalism can wait.

      4. Data as a tool for independent assessment: It is empowering to work with raw data and not relying solely on the findings of government committees, think tanks and academics, especially when the data is easily available, the analysis work is easy and the issue is hot in the news cycle. It also allows journalists take a fresh look at things and have more informed discussions with domain experts about their work, understand the analytical frameworks that form the backbone of their arguments, and get to the core of disputes when experts disagree on the interpretation of the same data. Doing this work made me appreciate the limitations of data work and a more mindful consumer of numbers in a world where a high-degree of bullshit comes packaged with charts and figures.

      5. Context is king: Every data analyst knows this. Having data is not enough. You need to ask interesting and meaningful questions to interrogate the data, place it in context, devoid of which you get meaningless number-heavy stories produced for the sake of filling newspaper pages. That's also the reason I enjoy collaborating with reporters who have strong domain knowledge of their beats.

      6. Data journalism is not objective journalism: Data is objective? Haha. I can narrate multiple stories from the same underlying dataset. It's easy. Context introduces subjectivity in interpretation, from questions you ask to what you consider significant. Data journalism should not occupy any special place in the larger philosophical discussions about objectivity in journalism. The same arguments are applicable.

      7. Data is not the solution to misinformation: Many solutions aiming to solve the internet-driven deluge of misinformation ignore how communication technology has evolved over centuries, how mediums impact how we learn about the world, the cognitive processes that drive how humans process and consume information, and the complexities of our chaotic and polluted information ecosystem. Ignoring that leads to an incomplete diagnosis of the problem and diverts attention to misguided solutions. "Data journalism as an antidote to fake news" is one of those.

      It is not. We are confronting a much larger epistemic problem and data-based articles are not going to solve it.

      8. It's important to know what you know and don't know, what you can know and can’t know: numbers, many think, offer divine certainty. In the wake of controversial issues dominating the news cycle, data journalists are asked to wave their magic wands to offer authoritative data-backed answers.

      There is some merit in that expectation. It is, in fact, the fundamental premise of evidence-based journalism. But it's not always possible and we must acknowledge the limitations of data. Reasons can vary from lack of data to important caveats which can't simply be left as an asterisk.

      Maybe the best answer an analysis can produce is: "we don't know for sure what the hell is happening. But we do know that A and B are likely false, C and D may be true, and not enough is known about E and F".

      This is a perfectly reasonable outcome of a meaningful data analysis. But this doesn't satisfy our deep desire for instant explanations and catchy headlines. In some cases, I see that a version of the story I skipped and filed in my trash—because it was not meeting rigorous standards—appeared in another publication. They let it through. This is especially true for reporting on non-scientific public opinion surveys. There is not much I can do about it, other than adding it to my long list of rants on how journalists do their work.

      9. Supporting data work with non-data work, and vice versa: A solid authoritative story is one where your key point is backed by quantitative data, qualitative evidence and theoretical arguments. If you find contradictions, the easy way is to leave it there, and write a version based on the "he said, she said" reporting template. The other option—which requires more work, and time—is to dig in further to reconcile what is happening and explaining those contradictions. If the data is good enough, it must reflect on something real that is happening.

      10. Data as an entry point into a story: A key journalistic lesson I have learnt in the last two years is that getting access to a novel dataset—also, documents—is not the end. Great stories emerge when you look at data as a lead in to a story, and not the story itself. Think about data as a human source, who came and told you something interesting, which makes you curious and raises questions, and you go on with a reporter's mindset to find answers. This is the data journalism I would like to see more of.

      11. Newsroom collaborations: Many journalists treat data reporters as members of a data desk, and say things like "I am working on this article, can you give me some data?" or "Can you make a chart for my story?".

      I haven't yet figured out polite ways to tell them this is not our job, and this is not the way to collaborate.

      Some journalists are just lazy, not even making the effort to find relevant data reports or call their sources to ask for directions, thinking they can just dump it on the data team. They are the ones who think of numbers and charts as an afterthought, a thing to strike off their to-do list.

      Some make the effort and reach out for help when they get stuck. A few see our skills as complementary and want to collaborate in telling a good story, right from ideation to production. These are folks I like to work with.

      12. Correlation is not causation: If you have read this far about data journalism, you have probably heard this phrase a zillion times. Yet, analysis pieces continue to attribute cause-effect relationships based on correlations alone. Correlations are important as they help in generating a possible set of hypothesis. But we must be careful in drawing inferences based on correlations.

      13. So what is causation? I also see a reverse trend: People say "correlation is not causation" and feel smart. Yet, many of the same folks will casually make causal claims in other places without hard evidence.

      Lack of causal explanations hinders our ability to answer important "why" questions. But thinking about causation, if done methodically and rigorously, is not trivial. I don't have any meaningful insight to offer here than just highlighting that this is also a problem I confront and currently in the process of thinking and reading more on causality.

      14. Lack of transparency: One of my major concerns that goes in tandem with my excitement looking at the rise of data journalism is the lack of transparency and openness—in sharing raw data, conveying analytical techniques, assumptions and calculation methods used in the analysis process.

      Editors don't have the time or ability to question the methods that end up generating a summary table. And readers don't have the relevant details or access to raw data to replicate the findings or test the conclusion for themselves. This is problematic.

      That's my general observation. There are exceptions: for instance, Mint does a good job of adding explanations on their calculations in chart descriptions. But I think as a community, we can do a better job of being more open and sharing more data. At the Hindustan Times, we started a GitHub repository for making all our data public, but the practice that didn't last beyond a few stories. There is no incentive apart from individual motivation. But it's not that hard if we place it high on priority. We should.

      15. Editorial filters: A common problem across newsrooms is the lack of senior editors who understand data. Every reporter—irrespective of how good or experienced they are—needs an editor who looks critically at their work. An editor's job is an important part of the machinery that makes journalism different from other forms of communication. But if editors are not equipped to edit and question data stories, the filter becomes weak and garbage is guaranteed. I am not immune to this either and guilty of having let some bullshit through in my writing—sorry about that—retrospectively hoping that an editor had spotted that stuff at the time of publishing. 

      16. Data literacy in the newsroom: While I always wish that more journalists appreciate the value in learning spreadsheets and put consistent efforts to get used to it, there is a deeper problem—the lack of data literacy. We don't emphasise enough on empirical thinking as one of the foundations for understanding the world when we talk about data in journalism. You don't need any number-crunching skills to find relevant data for a story. What comes first is asking the right empirical questions and having a frame to evaluate claims.

      Data literacy is different from data analysis. It is a skill in itself. You don't need a math degree to assess the quality of data and think about possible errors and biases in sampling or measurement process, to gauge the limitations of proxy stats, among other things. For instance, if someone sends you a survey report or a press release, you need to run basic sanity checks before referring it in your story.

      Data often misleads. To guard themselves, journalists need to become data literate and regularly fine-tune their bullshit detector.


      I repeat: all journalism should be data-informed. If you are a journalist interested in gaining a better understanding of data, I strongly recommend the following books:

      It's never too late to start learning new things!

      Questions or comments? I am at samarthbansal@protonmail.com. For occasional updates about my work, sign up here: bansalsamarth.substack.com

      #2: Why the event-oriented structure of news doesn't help in understanding how the world works

      In 2020, I significantly reduced the proportion of daily news consumption in my information diet. And I strongly recommend the same to others: less of news and more of books. There are many reasons why, and I will list them in a future post. Here is one compelling argument from the book Thinking in Systems (by Donella H. Meadows) about the fundamental limitation of incremental news stories:

      Systems fool us by presenting themselves—or we fool ourselves by seeing the world—as a series of events. The daily news tells of elections, battles, political agreements, disasters, stock market booms or busts. Much of our ordinary conversation is about specific happenings at specific times and places. A team wins. A river floods. The Dow Jones Industrial Average hits 10,000. Oil is discovered. A forest is cut. Events are the outputs, moment by moment, from the black box of the system.

      Events can be spectacular: crashes, assassinations, great victories, terrible tragedies. They hook our emotions. Although we’ve seen many thousands of them on our TV screens or the front page of the paper, each one is different enough from the last to keep us fascinated (just as we never lose our fascination with the chaotic twists and turns of the weather). It’s endlessly engrossing to take in the world as a series of events, and constantly surprising because that way of seeing the world has almost no predictive or explanatory value. Like the tip of an iceberg rising above the water, events are the most visible aspect of a larger complex—but not always the most important.

      We are less likely to be surprised if we can see how events accumulate into dynamic patterns of behavior. The team is on a winning streak. The variance of the river is increasing, with higher floodwaters during rains and lower flows during droughts. The Dow has been trending up for two years. Discoveries of oil are becoming less frequent. The felling of forests is happening at an ever-increasing rate. The behavior of a system is its performance over time—its growth, stagnation, decline, oscillation, randomness, or evolution.

      If the news did a better job of putting events into historical context, we would have better behavior-level understanding, which is deeper than event-level understanding. When a systems thinker encounters a problem, the first thing he or she does is look for data, time graphs, the history of the system. That’s because long-term behavior provides clues to the underlying system structure. And structure is the key to understanding not just what is happening, but why.

      #1: Why this blog

      To share:

      1. Notes, thoughts and ideas about new things I am learning

      2. Perspective on how I look at the world

      3. Nuggets from my reporting which did not make it to published articles

      4. Links to interesting things on the web—articles, research papers, videos, podcasts—and what I learnt from it

      5. To think in public. I will write about the uncertainties I battle with and the missing parts in my understanding.