I am a journalist and a programmer. I share raw thoughts, ideas and experiences on this blog. For occasional updates about my work, sign up here: bansalsamarth.substack.com
8342 words
samarthbansal.com@PySamarth

#6: Journalists refer to themselves as storytellers. Is that a mistake?

"Facts come in the way of beautiful prose," a colleague at the Wall Street Journal told me during my 2018 fellowship at the newspaper. 

It stayed with me. Every time I ponder over the limitations of journalistic methods to understand the world, I think about it. Especially in the last two years as I switched gears to work on more ambitious stories (relative to what I was doing earlier) with no easy answers. Even after tenaciously chasing all possible sources of information (data, documents, interviews, digital trails etc), piecing together a definitive story—say about a specific incident, a policy decision, an organisation—is hard. Very hard. 

How do you know for sure? 

In most cases, I operate on Bayesian probabilities. For an open-ended question, I start with a prior—my hypothesis on what I think is happening. Then I report. The new information moves the needle. It strengthens or weakens my prior. I try finding counter evidence for my hypothesis to stress test it. As reporting continues, some pieces start falling into place. The degree of belief for a few specific things starts increasing: most evidence points in the same direction and alternative explanations are proven wrong. 

Yet, the uncertainty remains. What have I missed? What is it that I am not looking at? These conflicts will remain and it is okay. It is an integral component of any knowledge production process. 

But things get tricky when you sit down to write. The most readable and enjoyable stories follow a clear narrative arc. Take fiction writing. The world being described exists inside the mind of the writer. They know everything: they are only bounded by their imagination. 

True stories are different. Journalism is different. Which is why I sometimes feel conflicted about the promise of narrative non-fiction genre. Won't facts and uncertainty play spoilsport when one is writing novelistic narratives with cinematic scenes? How do you handle that? 

This remains an open question for me and I was thinking about it last weekend. I wrote to my former WSJ colleague (referred above) to get his thoughts. His response was insightful and thought it's worth sharing. Here is our email exchange:

1. From me:

I have thought a lot about this over the years and it is a bit worrying. Almost all deeply reported stories have that defining paragraph (nutgraf) to signal why this story matters and why should readers spend their time on it. But many times, I see writers making those "larger points" that sound perfect in theory, but they may be stretching a bit beyond what the reporting material is revealing. There remains a subtle threat of overselling.

So I constantly remind myself that it is okay to not have that amazing paragraph (as much as I would love having it) where all words neatly add up to delight my readers. The nuances, the ifs and buts, matter. However, so many journalists feel they know-it-all and write with such great authority that it automatically lends itself to beautifully written prose.

Is this conflict widely acknowledged in the American press? I would love to read more on what editors think about it. 

2. From my former WSJ colleague:

I'm not sure that there has been any great introspection on the part of American journalists when it comes to these issues. I think Tom Wolfe, Truman Capote, and Gay Talese helped lead journalists astray with their riveting, but dubiously sourced, "New Journalism." It was exciting and eminently readable, but couldn't possibly be totally true and correct.

I often hear journalists refer to themselves as storytellers, but I think that is a huge mistake and leads to the kind of overreach and crafting that you mention. We should see ourselves first as fact relayers, and we should be open and honest about what we do and don't know and why our "story" doesn't follow a perfect arc.


We should clearly source everything. We should clearly and generously credit those whose work informed ours.We should always understand that the "why" part of the five w's is the one that we can never be close to sure about. How do we presume to get inside an actor's head and understand his motivations, or that we can account for hundreds or thousands of random events and people that come together at a moment in time and lead to something happening?

I always have these thoughts when I read foreign correspondents who have parachuted into a foreign land and confidently assert the cause-and-effect of endlessly complicated situations. I would never have the self assuredness to do that.

I love the term sophomoric, because we all go through that phase of learning so much that we think we know it all. One nice thing about growing older is that you really have learned a lot, but you also have learned just how much you don't and might never know (if you're paying attention at all).


#5: How I approach and manage my freelance journalism career

"There is a lot of uncertainty in figuring out how this will work out," reads the seventh point in my journal entry titled "On Quitting", dated 13th January 2019—the day I resigned from my full-time job at the Hindustan Times

After two excellent years at HT—where I got the chance to learn from and collaborate with a bunch of talented and hardworking journalists, where editors gave me the opportunity to hone my craft, space to write meaningful stories and nominated me for an amazing fellowship at the Wall Street Journal—I was done with it.

As in most jobs, my stint at HT had both upsides and downsides. I struck a balance, making most out of the opportunities at hand, downplaying stuff that troubled me, and moving on. But everything has a threshold. Somewhere around December 2018, I knew it was time to leave. I was not fitting in. If I wanted to stay inspired to be a journalist, I had to quit. And so I did. 

At the time of resignation, I had no plan for what comes next. "How long? Who will read? Who will pay? What next? Am I in the right profession?" I wrote in my journal. "These questions come naturally when you know that salary won't be automatically credited to your account," I added, anticipating my cash flow in the coming months. "It does not sound ideal. But at least it is forcing me to think about my priorities. And that’s good."

Two years later, with the benefit of hindsight, I am so glad I took that call to quit my job. I did not go back to a full-time role at a news organisation—by choice, not due to lack of opportunities. I now work as a freelance journalist: In 2019, I exclusively did reporting and writing; in 2020, I started with a dual career, splitting time between freelance journalism and software engineering. 

I love working independently. I earn enough to pay my bills, save and invest; I am getting closer to the kind of work I aspire to do; I get a hell lot of time to learn new things, read, relax, pause, sit and just think; most importantly, I can prioritise stuff I had long ignored in early adulthood: my physical and mental health.

In the last two years as a freelance reporter, I have written long stories for various Indian and International publications. You can read my published work here

I am writing this post to share my experience. 


1. Background: We are not playing the same game

I hesitate to offer freelancing advice. Consider this: When I quit my job, I was 25, had a year's worth of emergency fund, no debt, no EMIs, and because I live with my parents, no rent to pay. That means, even if things would have gone downhill, and nothing worked out, I could still comfortably get through another year. This cushion allowed me to wait and see how freelancing goes, rather than immediately look for a new full-time job. 

Add to that the two intangible assets I own: an elite undergraduate degree (mathematics and scientific computing from IIT Kanpur) and high-in-demand technical skills (building software, data science). I think of them as insurance instruments to protect me from any future shocks. In the winner-take-all economy—which favours the elite and perpetuates inequality—my educational background and interests have equipped me to enter that elite group. (There is a great deal of discomforting dissonance between who I am and the vision of the society I want to live in. But it is what it is.)

So simply put, I have a lot of structural privilege. I am fortunate to not have to live with survival questions. If all hell breaks loose, I always have the option to build a great career in the technology industry. I can't stress enough how much this matters in every single thing I do. 

I mentioned this to make one thing clear: my worst-case scenario is very safe. As much as leaving a job without a plan sounds like a risk, it wasn't a big deal: the stakes were low. 

It is not so for most people.

Moreover, multiple factors will dictate your experience: social capital and network (people you know), skills (what you know, what you can do, how you fare with competition), random chance events (on which you have no control), your expectations from life and how you perceive risk.

It is important to keep this context in mind as you read further about my experience. I have figured out a model that works for me — at least for now. But no two people can have an identical model: we are not playing the same game. There is no right approach. It varies with who you are and what you want. What worked for me may only be partially applicable to your scenario, or perhaps not at all. 

This post only describes my experience as an individual. 


2. Framework: What do you want?

Goal clarity is the single most important thing to deal with the inevitable uncertainty of a freelance career. The future is unknowable, and plans rarely work — beyond six months, no rigid plan has ever worked for me. So my approach is to operate within self-imposed rules that align with my larger abstract goals. They guide my decisions and help me choose. Especially so in thinking about trade-offs, and in rejecting lucrative opportunities with a high short-term appeal. 

Most of my goal-setting process revolves around having clarity on five sets of questions:

1. Why am I a journalist? Why not any other profession? What do I expect from my work?

2. What kind of journalism I want to do? What is that I do not want to do?

3. Why am I freelancing? Why not find a full-time job at a publication? What can I do in a big media setup that I can’t do independently, and vice versa? What are the trade-offs involved?

4. What are my needs and wants, right now and a little ahead in the future? How much money do I need to fulfil those? What is my personal finance philosophy?

5. What kind of life do I want outside work? What are the things I value in my life? What are the things that matter to me? What are my core underlying principles? 

These are hard questions. I struggle answering them. They force me to ask deeper questions about my belief systems, most of them hidden from immediate view, wrapped in dominant ideologies of the day. But over several months, scribbling in my journal, I figured where I stand at the moment, helping me set clear personal and professional goals. Of course, these answers are not set in stone and will change with newer life experiences. 

Here is a brief glimpse: I don't find it thrilling to get a lot of bylines. I enjoy writing long stories, exploring one topic in depth. I can't recklessly follow the news cycle. I prefer working with good editors who help me improve my craft than getting published in a big publication for the sake of it. 

I don't care about the "impact" of my work—I have no control over that. But I do have control over effort and integrity, and it matters to me. Reporting rigour in stories should follow an upward trajectory. I should learn new things. I value journalistic freedom more than anything else, even if that comes at the expense of readership. 

I am incapable of maintaining an active social media presence, especially Twitter. High-quality leisure is important, and I don't want my work to take over everything else I care about. Ethics and principles remain at the centre of my decision making, and I try to reduce the inescapable moral contradictions we all live with to whatever degree possible. 

    The list goes on. 

    Freelancing allows me to curate my career to fit many of these preferences and gives me more control over my time. That's the reason I stuck with it. 

    I did apply for one full-time job in the last two years. A couple of months after I left HT, the Wall Street Journal had an opening for a cyber-focused investigative reporter and an editor at the paper—who mentored me during my 2018 fellowship at the Journal—suggested I should check it out. I applied, the process began, but it didn't work out. I didn't apply for any full-time position after that. 

    I do have a few dream jobs and at some point in my career, I may try getting there. Not now, though. It doesn't feature in my six-month goals—the maximum time I plan for. 


    3. System: How to achieve your goals?

    Goals are crucial. Equally important is to have a pragmatic system to take you there. This includes the specific mechanics of getting assignments, managing money, balancing multiple projects and meeting deadlines. 

    Here is how I get reporting gigs:

    1. I write infrequently and mostly long-form reported stories. There are many publications interested in original reporting, and I am lucky that story ideas that I find exciting overlap with the interests of many publications which have budgets to hire freelancers. 

    2. Sometimes, an editor approaches me with an idea, or just an offer to contribute to their publication. In other cases, I pitch ideas to specific publications, checking if they would be interested. It goes both ways. 

    3. I started freelancing after working at India's largest publicationsThe Hindu and Hindustan Times—for three years. So I knew a couple of editors who hire freelance writers which made it easy for me to get started. 

    4. What has worked the best: building solid relationships with a few editors. Keeping in touch about ideas and writing for them once in a while. These editors have a good understanding of how I work and what they can expect if they commission an assignment to me. I strongly believe that the key variable predicting a journalist's output over the long run is their reporting process, everything from ideation to writing. I have been working hard on improving my process—a work in progress—to tackle complicated stories. Editors who know me see that in action and that sets the right expectations for a mutually beneficial arrangement. 

    5. I like to think of myself as a beat-agnostic reporter: not restricting my reporting to a few themes and chasing stories that pique my curiosity. That's ideal. But it's good to have a niche that defines why an editor should reach out to you. In my case, you will see three strands: reporting at the intersection of technology, politics, policy and culture; long-term investigations; and data journalism. This combination has limited supply in a crowded journalism market and helps me in getting assignments. 

    6. I always ask myself: If I get to tell a story, what is the value I am bringing in? What will make my story stand out if five different reporters are simultaneously working on it? How can I be authentic? If editors can see that value, the chances of getting another assignment increase. 

    In the last two years, I have not seen a phase where I had to struggle to find work. I hope it stays that way. 

    On balancing projects/timelines: I am not good at this. Things keep piling up, the todo list keeps extending, leading to weeks where I have to continuously slog to wrap things up. I am working on this and learning how to manage my time better. 

    Financial management deserves a detailed discussion. 


    4. Money: How do you sustain?

    "Huh, so you are living off your parents' money," said a senior journalist, whose work I admire, when I mentioned I work as a freelancer. I would not be surprised if many others believe the same. I just wish they also factor in that journalism is not the profession most Indian parents would willingly cross-subsidise. Not mine, especially when their kid has a STEM degree. The day I graduated from college, I knew one thing: the only way to resist parental pressure is to remain financially independent. 

    But I understand why they think so. The most common question friends ask me is about money. "Do you get paid enough?", "How do you sustain?", "Isn't it risky?"

    Let me turn the table for a moment: Do you feel secure at a large media organisation? What about layoffs? How do you feel when your annual raise is less than the rate of inflation? 

    The point is simple: you can't escape risks. But you can find ways to manage them.  

    Look, I don't want to make a killing fortune — and if you do, please don't become a journalist. Wrong profession for that goal. But neither do I want to be miserable at any point or worry about my finances all the time. I don't have too many material desires, but I am no monk either. What I need is a way to strike that fine balance. 

    1. The question that drives my thinking around finances: how I can do what I want to do while earning enough? Being a freelancer is like running a one-person business. I do not care about the larger economic crisis in journalism on a day to day basis: all I want is a publication willing to pay me, fairly, for my next assignment.

    2. My strategy: I start by asking how much I need. I assign myself a modest annual package—Rs X lakhs per year, just like a salary at a regular job. With the amount set, I work backwards: what do I need to achieve that X? Then I fill the bucket as much as possible with earnings from the stuff I like to do. For the remaining sum, I don't care as much about the specifics of the work: I just need to fill the bucket to make my system work. 

    3. Finances in 2019-20: This approach worked out pretty well for me. In the first year, I made almost all my money exclusively through writing assignments. That's what encouraged me to continue working as a freelancer. To give you a sense, in the first year of freelancing, I set myself an annual target equal to 90% of what I was making at the Hindustan Times. 90% because I did not want to get too ambitious about finances right in the first year, as I was figuring out how all of this works. But neither did I want to keep it too low and move forward with an unrealistic evaluation.

    4. Payment structures: Many times I have been paid similar per-word rates for a story I wrote in one day and another that took me three weeks. It makes no sense, but many publications only operate on word-count based rates and you can't do much about it.

    Some publications pay you for your time. For instance, in 2020, I signed up for a two-month exclusive arrangement with HuffPost India where I would be paid a monthly fee, irrespective of the number of stories or their length. I had a similar contract with the India Today group, also for a couple of months, to cover the 2019 general election. 

    5. Cross-subsidise one assignment with another: Foreign publications pay significantly more than Indian publications—even though the effort might be similar. A higher payment from one assignment enables me to spend more time on relatively lower-paid assignments. 

    6. Rates: the per-word rates in Indian publications average around Rs 10-15 (good ones – some pay even less than Rs 5 per word or nothing at all). American publications pay in dollars, and I have seen rates ranging from 0.5 to 1.5 USD. For example:

    PublicationRates
    HuffPost India 
    (not operational any more)
    Rs 10 per word/custom rates for special assignments
    Mint Long StoryPre-pandemic: Rs 30,000 (~2000-2200 words)

    Now: Rs 20,000 (for same stuff)

    The Atlantic$0.5 per word (~Rs 37 per word)
    Rest of World$1.5 per word (~Rs 100 per word)


    7.
    Delayed payments: A quick Twitter scan shows me delayed payments are a norm in the industry. Fortunately, I have almost always been paid in time. My first payment from Mint was delayed by a few months — but that was the last time it happened. 

    8. My side job:
     Many freelancers I know have some sort of a side job, leveraging their core skills elsewhere. For instance, a journalist friend splits her time between reporting/writing (core work) and working for a big data startup (side job) who hired her to write stories based on insights from their analytics for their internal use. A freelance musician I met at a cafe splits his time between producing music (core work) and teaching music at a university (side job). And so on.

    The idea is to diversify your income source and not be dependent on writing alone. I got on a side job mode in my second year of freelancing, leveraging my programming skills. 

    In January 2020, a friend and I started PopperAI, a news automation company. Our software created thousands of machine-generated articles based on high-frequency public data. But we shut shop in July due to various reasons. The purpose of starting the company was clear: it was a means to an end. I wanted to build a parallel high-value revenue stream for myself which would buy me more time for reporting ambitious stories. Alas, it didn't work out and my financial model for 2020 took a hit, albeit temporarily. 

    I then explored other freelance opportunities in software engineering and got a six-month contract with a Bengaluru-based startup, where I now contribute part-time. It's a great deal, especially because the company gives me the flexibility to work on my schedule. 

    I find the side job very empowering: it drastically reduces uncertainty and helps me focus. The fact that I enjoy writing code and thinking about engineering problems makes it the best of both worlds. 

    9. Building a healthy and thoughtful relationship with money: carefully thinking through my personal finance philosophy helped me a lot. Maintaining a low burn rate and a high savings rate, creating an emergency fund, planning for short-term/long-term investments, buying a good health insurance policy—all the standard things people recommend to make money work for you. Consider reading Morgan Housel's book Psychology of Money and Monika Halan's Let's talk money to educate yourself. It's so important. It took me some time to get in that flow. 


    5. Other things: what is it like to be a freelancer?

    - Where I work: I have been working from home long before the pandemic made it a compulsion. But I am more productive when not at home. In normal times, I work from coffee shops. There are several work-friendly cafes in New Delhi and I identified a few spots where I am a regular — who let me sit through the day even if all I order is one cup of coffee. I considered moving to a co-working space at one point. A flexible seat costs around Rs 8000 per month. But I decided against it and stuck with cafes. 

    - Become a digital nomad: If you are a nature person, go work from hills. It's amazing. Places like Zostel Mussoorie—one of my favourite spots—are designed for remote working. This will only grow post-pandemic as both the supply and demand are likely to increase. 

    - You have to hold yourself accountable: In a regular job, you are told what your mandate is. You have an editor who has expectations. You have a fixed salary. Freelance changes that. You have to decide everything for yourself. You get to decide what you want to do and how. You can direct your career to the style of journalism you prefer. You decide the extent to which you will report a story. You decide the bounds. The freedom can be daunting if not leveraged thoughtfully. It's not easy.

    Work is no longer the central part of my life: The institutions we are part of and the work we do there become a part of our identity. That erodes in many ways as a freelancer. There is no concept of weekdays or weekends. Every day is different. For instance, if I am desperate to finish a new book, I can just not work for two days and read. Or meet friends in the afternoon and work late in the evening. Then there are weeks where I am packed with assignments and have to work every day. I can plan a two-week trip to the mountains and then take more work when I come back. I love this flexibility.

    - You only have control over your own story: When employed at a news organisation, I think beyond my work. I care about the newspaper I am part of and try contributing, in small ways, to improve the overall coverage of the publication. On big event days, I loved collaborating with the team. I could raise red flags about the work we do with folks who make editorial calls. In short, you can contribute to building stronger institutions. Freelancers don't get to do that. You operate as a solo journalist and all you can do is do a good job on your article. 


    I hope this was helpful for you. Questions or comments? I am at samarthbansal@protonmail.com. For occasional updates about my work, sign up here: bansalsamarth.substack.com


      #4: How interlinked economic and political forces create self-censorship in Indian media

      Most contemporary discussions on press freedom begin with some sort of rankings: X country slipped Y positions on Z index—that’s evidence something wrong is happening.

      This makes headlines every year in India, as we continue to slip down in these indices. Criminal defamation cases are filed against journalists for merely doing their jobs and reporting stories. Organized disinformation campaigns and troll armies delegitimize the institution of journalism and individual reporters to evade trust in the media. The list goes on. These observable metrics—all problematic—reflect in the various rankings.

      But I don’t find much meaning in this data. As someone who plays with numbers for a living—I am a data journalist—I remain deeply skeptical about any methodology that claims to quantify press freedom. It’s hard. That’s because the everyday experience of being inside a newsroom, the anxieties journalists live with and forces that shape editorial decisions stay hidden from the outside world — all they get to see is the output, which, however egregious, doesn’t capture the true extent of the rot in the Indian press.

      Let me give you a brief glimpse of a few revealing aspects of the big media. I will restrict myself to English language publications with the largest readership and circulation.

      Of India’s four largest English newspapers — Times of India, The Hindu, Hindustan Times and Indian Express — only Express has a dedicated investigative reporting team, which, for instance partners in international stories like Panama or Paradise papers. Their team strength: less than five.

      None of the other three papers even have the position of an “investigative editor”. That does not mean the papers don’t investigate or publish critical stories — or, as the public opinion goes, “sold out” to the government. It’s not that simple. Their coverage is crucial to our understanding of modern India and reporters at those papers are doing important work. But the point is, hard hitting stories are rare, and when they appear, it’s driven by the efforts of a few individuals pushing boundaries rather than a systemic culture encouraging publication of uncomfortable stories. Don’t judge a paper by what it publishes. What is excluded is generally more telling.

      In September 2018, I returned to India after spending five months reporting with the investigative reporting team of the Wall Street Journal in New York as part of a fellowship program. I was pumped up to use my newly acquired skills for reporting in India. However, on the very first day, when I started meeting journalist friends, I saw a drastic cultural difference: at the Journal, you pitched an investigative idea, and colleagues helped me brainstorm how to report the story; in India, colleagues said no point chasing this story as it won’t be published.

      That’s not without reason. Consider this: I reported a story showing how propaganda is disseminated and troll accounts are promoted on the Twitter-like interface in the official app of Narendra Modi, India’s Prime Minister. My editor sat on the story for two months. Why? He was busy. But he did have time to edit the other stories I was filing. No explanation was offered.

      In another case, I wrote a story that showed how the Twitter metrics — retweets per tweet, for instance — of India’s opposition leader Rahul Gandhi had surpassed that of Mr Modi. All hell broke loose for publication of such a benign story (it took me half a day to analyse and write this short story). Union Ministers from Mr Modi’s party accused my report of propagating partisan propaganda. Next day, I got a phone call from the owner of the publication—whom I had never met—asking me to follow up and check if the opposition leader’s metrics were amplified by a bot. I should send the story directly to the owner and not to the editor. Sigh. It is not too hard to imagine what would have transpired behind the scenes: actions that remain a mystery for reporters at my pay grade.

      This toxic culture of self-censorship has become oblivious: many journalists may not even recognize the boundaries they have constructed in imagining what a story could be. Once, I was given an assignment to analyse the content of the Prime Minister’s campaign speeches. The editor wanted numbers: how much time, as a proportion of the overall speech, does Mr Modi take shots at the opposition than talking about his promises? It was a fairly analytical story. I had to randomly select a few speeches, make sure I cover different time spans and states, listen to the speeches, create a dataset, and tell what I found. A senior political reporter was asked to help me. The first question the reporter asked: “have you been told what the analysis needs to say?” I was shocked. The implications were clear: it was normal for that reporter — at least ten years older than me — to be sometimes told what the story needs to say before reporting it. That’s not acceptable.

      It took me some while to understand how common this is: some reporters start doubting themselves and ask whether they are chasing stories that don’t matter, whether they have not done the job properly, or simply the institution they work for is compromised. The answers don’t come easily. Others, who don’t think too much about these supposedly “idealistic” questions, continue to flourish. “Not a single story I wrote was ever censored,” they will tell you. Of course—it won’t be.

      This form of self-censorship, I would argue, is a bigger problem than explicit censorship. Because it happens subtly and parasitically cripples the institution of journalism. It’s driven by interlinked economic and political forces. For instance, most of India’s large media houses that are struggling with cash organize grand annual conclaves to bump up their bottom lines: Political leaders, top company executives, sport stars, and Bollywood celebrities are invited as guests. The bigger the names that speak at your conclave, it is said, more money sponsors put in.

      But what if Mr Modi is in your speaker list, and he refuses to turn up at your event at the last moment? It could be disastrous for the company’s finances. This is not a theoretical case: it has happened with one of India’s largest papers. “Security concerns” was the official reason, but news reports flagged the PM was not happy with the paper’s coverage. Which simply means you can’t offend Mr Modi’s sensibilities for at least a month preceding the event. Imagine the pressures a reporter would face to even pitch a story that holds the PM accountable.

      I have deliberately chosen all the examples that relate to Mr Modi. In journalism circles, it is often said: you can criticize the government, write about his party, but things get complicated when the story centers around the Prime Minister himself.

      George Orwell wrote: “Intellectual honesty is a crime in any totalitarian country; but even in England it is not exactly profitable to speak and write the truth”. With all signs of India sliding towards an illiberal democracy, the situation here is no different.

      So, hypothetically speaking, even if India’s position in press freedom rankings jumps a few places, it would be no cause to celebrate. Unless the systemic problems which one finds inside India’s largest newsrooms begin to change, and reporters and editors can think and write freely, while being committed to the craft and core values of the profession, free press in India will remain a myth.

      Originally published at n3Con magazine

      #3: Thoughts and observations on data journalism in India

      Today, I complete five years in journalism: one year at The Hindu (my first job, straight out of university), two years at the Hindustan Times—which overlapped with a five-month fellowship at the Wall Street Journal—and two years as a freelancer.

      "Data journalist" was my official job description in full-time roles. But I find that term redundant and I don't like to use it anymore: all journalism should be data-informed and evidence-based. I prefer calling myself just a "reporter". In part, this identity shift—primarily in my head, I doubt if anyone else cares—is a reflection of the changing nature of my work: I have moved from writing newsy analytical stories to a mix of investigative and narrative reportage. Data and empirical thinking continue to be at the centre of my work, but it's no longer defined by it.

      This post is about data journalism in the way most people understand it: journalists who specialise in producing data-driven stories, a task that includes everything from collecting data to analysing and presenting it visually.

      Here are some thoughts and observations on data work in Indian newsrooms.

      1. Data journalism is an umbrella term which means different things to different people: adding data-backed context in daily news coverage; analysis of issues dominating the news cycle; making infographics with interesting data factoids; narrating stories and explaining complex concepts using visual graphics; using hard data for investigations, highlighting previously unreported trends and throwing light on specific incidents; empirical coverage of big-ticket events like elections and budgets.

      I have done all of this. It's useful to think of them as different tasks as the skillsets vary widely. A lot of stuff can be done without writing a single line of code or making pivot tables in spreadsheets. But the fanciest stuff we do gets the most attention, making regular data work appear more complicated than it is.

      2. Data journalism is not just about data visualisation or interactive stories: I categorically mention this as a separate point because I used to conflate the two when I started. Back in 2015, data journalism for me was the visual stuff the New York Times published. I would browse their interactive stories, think about how we can take inspiration for telling similar stories in India, rant about the technical restrictions our archaic online publishing systems posed, and ultimately make peace with the baby steps we took in creating an ecosystem that valued interactive storytelling.

      I learnt my lesson soon: I was getting too attached to the form. Yes, charts are nice. Interactives, when thoughtfully designed, can help tell brilliant stories. But that should not divert our focus from the fundamentals, which is the data itself and the meaning it's adding to the story. Once you get past that stage, you can figure out how to stitch words and visuals to communicate your findings to the world. Sometimes, one small static chart can tell a story that no words can capture, and in other cases, data leads you to nuanced explanations where clear prose becomes crucial.

      3. Visual journalism in India can wait: If a newsroom has infinite resources, sure, they can—should—hire a team of a dozen or two coder-journalists with excellent programming and design skills. Let them work on exciting special projects, fully leveraging the interactive nature of the internet to tell relevant and compelling stories. From mid-2016 to mid-2018, our small data and visuals team at the Hindustan Times—synonymous in journalism circles with bylines of Gurman Bhatia and Harry Stevens—did just that. I loved the work we did, and I learnt a lot. Bhatia has since moved to work for Reuters in Singapore and Stevens is at the Washington Post.

      But over these years, as I have been thinking more broadly about Indian journalism, I have become increasingly convinced that we are not yet ready to institutionalise this kind of work. Cash-constrained Indian newsrooms neither have the resources to hire that talent nor the imagination to nurture it. Yes, things have to start somewhere, but I don't see that time is now.

      Instead, we should be prioritising work that produces original reporting and boosts collective news-gathering efforts. Newsrooms should hire journalists who can code and ask them to think creatively and build original datasets, ideate on innovative ways to use statistics in identifying ignored trends, invest in long investigative projects. While the focus remains on sharp data reporting, newsroom nerds can occasionally produce visually appealing journalism.

      Journalists should still make simple, effective and aesthetically pleasing charts using simple tools like DataWrapper, which have evolved significantly in terms of capabilities over the last few years. But for now, a dedicated focus on building a culture of interactive and visual journalism can wait.

      4. Data as a tool for independent assessment: It is empowering to work with raw data and not relying solely on the findings of government committees, think tanks and academics, especially when the data is easily available, the analysis work is easy and the issue is hot in the news cycle. It also allows journalists take a fresh look at things and have more informed discussions with domain experts about their work, understand the analytical frameworks that form the backbone of their arguments, and get to the core of disputes when experts disagree on the interpretation of the same data. Doing this work made me appreciate the limitations of data work and a more mindful consumer of numbers in a world where a high-degree of bullshit comes packaged with charts and figures.

      5. Context is king: Every data analyst knows this. Having data is not enough. You need to ask interesting and meaningful questions to interrogate the data, place it in context, devoid of which you get meaningless number-heavy stories produced for the sake of filling newspaper pages. That's also the reason I enjoy collaborating with reporters who have strong domain knowledge of their beats.

      6. Data journalism is not objective journalism: Data is objective? Haha. I can narrate multiple stories from the same underlying dataset. It's easy. Context introduces subjectivity in interpretation, from questions you ask to what you consider significant. Data journalism should not occupy any special place in the larger philosophical discussions about objectivity in journalism. The same arguments are applicable.

      7. Data is not the solution to misinformation: Many solutions aiming to solve the internet-driven deluge of misinformation ignore how communication technology has evolved over centuries, how mediums impact how we learn about the world, the cognitive processes that drive how humans process and consume information, and the complexities of our chaotic and polluted information ecosystem. Ignoring that leads to an incomplete diagnosis of the problem and diverts attention to misguided solutions. "Data journalism as an antidote to fake news" is one of those.

      It is not. We are confronting a much larger epistemic problem and data-based articles are not going to solve it.

      8. It's important to know what you know and don't know, what you can know and can’t know: numbers, many think, offer divine certainty. In the wake of controversial issues dominating the news cycle, data journalists are asked to wave their magic wands to offer authoritative data-backed answers.

      There is some merit in that expectation. It is, in fact, the fundamental premise of evidence-based journalism. But it's not always possible and we must acknowledge the limitations of data. Reasons can vary from lack of data to important caveats which can't simply be left as an asterisk.

      Maybe the best answer an analysis can produce is: "we don't know for sure what the hell is happening. But we do know that A and B are likely false, C and D may be true, and not enough is known about E and F".

      This is a perfectly reasonable outcome of a meaningful data analysis. But this doesn't satisfy our deep desire for instant explanations and catchy headlines. In some cases, I see that a version of the story I skipped and filed in my trash—because it was not meeting rigorous standards—appeared in another publication. They let it through. This is especially true for reporting on non-scientific public opinion surveys. There is not much I can do about it, other than adding it to my long list of rants on how journalists do their work.

      9. Supporting data work with non-data work, and vice versa: A solid authoritative story is one where your key point is backed by quantitative data, qualitative evidence and theoretical arguments. If you find contradictions, the easy way is to leave it there, and write a version based on the "he said, she said" reporting template. The other option—which requires more work, and time—is to dig in further to reconcile what is happening and explaining those contradictions. If the data is good enough, it must reflect on something real that is happening.

      10. Data as an entry point into a story: A key journalistic lesson I have learnt in the last two years is that getting access to a novel dataset—also, documents—is not the end. Great stories emerge when you look at data as a lead in to a story, and not the story itself. Think about data as a human source, who came and told you something interesting, which makes you curious and raises questions, and you go on with a reporter's mindset to find answers. This is the data journalism I would like to see more of.

      11. Newsroom collaborations: Many journalists treat data reporters as members of a data desk, and say things like "I am working on this article, can you give me some data?" or "Can you make a chart for my story?".

      I haven't yet figured out polite ways to tell them this is not our job, and this is not the way to collaborate.

      Some journalists are just lazy, not even making the effort to find relevant data reports or call their sources to ask for directions, thinking they can just dump it on the data team. They are the ones who think of numbers and charts as an afterthought, a thing to strike off their to-do list.

      Some make the effort and reach out for help when they get stuck. A few see our skills as complementary and want to collaborate in telling a good story, right from ideation to production. These are folks I like to work with.

      12. Correlation is not causation: If you have read this far about data journalism, you have probably heard this phrase a zillion times. Yet, analysis pieces continue to attribute cause-effect relationships based on correlations alone. Correlations are important as they help in generating a possible set of hypothesis. But we must be careful in drawing inferences based on correlations.

      13. So what is causation? I also see a reverse trend: People say "correlation is not causation" and feel smart. Yet, many of the same folks will casually make causal claims in other places without hard evidence.

      Lack of causal explanations hinders our ability to answer important "why" questions. But thinking about causation, if done methodically and rigorously, is not trivial. I don't have any meaningful insight to offer here than just highlighting that this is also a problem I confront and currently in the process of thinking and reading more on causality.

      14. Lack of transparency: One of my major concerns that goes in tandem with my excitement looking at the rise of data journalism is the lack of transparency and openness—in sharing raw data, conveying analytical techniques, assumptions and calculation methods used in the analysis process.

      Editors don't have the time or ability to question the methods that end up generating a summary table. And readers don't have the relevant details or access to raw data to replicate the findings or test the conclusion for themselves. This is problematic.

      That's my general observation. There are exceptions: for instance, Mint does a good job of adding explanations on their calculations in chart descriptions. But I think as a community, we can do a better job of being more open and sharing more data. At the Hindustan Times, we started a GitHub repository for making all our data public, but the practice that didn't last beyond a few stories. There is no incentive apart from individual motivation. But it's not that hard if we place it high on priority. We should.

      15. Editorial filters: A common problem across newsrooms is the lack of senior editors who understand data. Every reporter—irrespective of how good or experienced they are—needs an editor who looks critically at their work. An editor's job is an important part of the machinery that makes journalism different from other forms of communication. But if editors are not equipped to edit and question data stories, the filter becomes weak and garbage is guaranteed. I am not immune to this either and guilty of having let some bullshit through in my writing—sorry about that—retrospectively hoping that an editor had spotted that stuff at the time of publishing. 

      16. Data literacy in the newsroom: While I always wish that more journalists appreciate the value in learning spreadsheets and put consistent efforts to get used to it, there is a deeper problem—the lack of data literacy. We don't emphasise enough on empirical thinking as one of the foundations for understanding the world when we talk about data in journalism. You don't need any number-crunching skills to find relevant data for a story. What comes first is asking the right empirical questions and having a frame to evaluate claims.

      Data literacy is different from data analysis. It is a skill in itself. You don't need a math degree to assess the quality of data and think about possible errors and biases in sampling or measurement process, to gauge the limitations of proxy stats, among other things. For instance, if someone sends you a survey report or a press release, you need to run basic sanity checks before referring it in your story.

      Data often misleads. To guard themselves, journalists need to become data literate and regularly fine-tune their bullshit detector.


      I repeat: all journalism should be data-informed. If you are a journalist interested in gaining a better understanding of data, I strongly recommend the following books:

      It's never too late to start learning new things!

      Questions or comments? I am at samarthbansal@protonmail.com. For occasional updates about my work, sign up here: bansalsamarth.substack.com

      #2: Why the event-oriented structure of news doesn't help in understanding how the world works

      In 2020, I significantly reduced the proportion of daily news consumption in my information diet. And I strongly recommend the same to others: less of news and more of books. There are many reasons why, and I will list them in a future post. Here is one compelling argument from the book Thinking in Systems (by Donella H. Meadows) about the fundamental limitation of incremental news stories:

      Systems fool us by presenting themselves—or we fool ourselves by seeing the world—as a series of events. The daily news tells of elections, battles, political agreements, disasters, stock market booms or busts. Much of our ordinary conversation is about specific happenings at specific times and places. A team wins. A river floods. The Dow Jones Industrial Average hits 10,000. Oil is discovered. A forest is cut. Events are the outputs, moment by moment, from the black box of the system.

      Events can be spectacular: crashes, assassinations, great victories, terrible tragedies. They hook our emotions. Although we’ve seen many thousands of them on our TV screens or the front page of the paper, each one is different enough from the last to keep us fascinated (just as we never lose our fascination with the chaotic twists and turns of the weather). It’s endlessly engrossing to take in the world as a series of events, and constantly surprising because that way of seeing the world has almost no predictive or explanatory value. Like the tip of an iceberg rising above the water, events are the most visible aspect of a larger complex—but not always the most important.

      We are less likely to be surprised if we can see how events accumulate into dynamic patterns of behavior. The team is on a winning streak. The variance of the river is increasing, with higher floodwaters during rains and lower flows during droughts. The Dow has been trending up for two years. Discoveries of oil are becoming less frequent. The felling of forests is happening at an ever-increasing rate. The behavior of a system is its performance over time—its growth, stagnation, decline, oscillation, randomness, or evolution.

      If the news did a better job of putting events into historical context, we would have better behavior-level understanding, which is deeper than event-level understanding. When a systems thinker encounters a problem, the first thing he or she does is look for data, time graphs, the history of the system. That’s because long-term behavior provides clues to the underlying system structure. And structure is the key to understanding not just what is happening, but why.

      #1: Why this blog

      To share:

      1. Notes, thoughts and ideas about new things I am learning

      2. Perspective on how I look at the world

      3. Nuggets from my reporting which did not make it to published articles

      4. Links to interesting things on the web—articles, research papers, videos, podcasts—and what I learnt from it

      5. To think in public. I will write about the uncertainties I battle with and the missing parts in my understanding.